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Preface 


Francesca Fatta 


When in January 2020 Roberta Spallone with Andrea Giordano and Michele Russo pro- 
posed a seminar with the complex acronym REAACH-ID to involve a larger group of 
Augmented Reality scholars, | realized that the time had come to define the context of our 
initiatives on virtual representation, looking also to the sphere of science that deals with 
Artificial Intelligence. Their proposal did not want to follow the trend, but rather to respond 
to the need for methodological relationships and cross competences aimed at promoting a 
dialogue between human and hard sciences. 

The starting point consists in the single and shared researches of the scholars involved in 
the organization of this symposium, they are all teachers of Drawing who have been en- 
gaged for years in the creation and analysis of drawings, images and models that represent 
the evolution of existing realities over time or purely designed entities. Moreover, when the 
talian teachers of Drawing reorganized the declaration of the sector (UID May 2021), they 
agreed that the scope of our SSD (Scientific Disciplinary Sector) research: “[...] therefore 
concerns the geometric-descriptive-configurative domains, graphic-visual-synaesthetic, in- 
formational-computational features, as well as the related historical, epistemological, seman- 
tic, technological and applicative aspects. [...] the modeling, including informative modeling, 
prototyping and visual communication [...]”. 

The question concerns the definition of the relationships between a physical, real, tangible 
reality and an intangible spatiality defined with the help of Artificial Intelligence, increasingly 
able to trigger specific reading processes of complex contexts, which can be represented in 
a way similar to human thought with amazing space-time simulation effects. 
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Mario Rasetti's prestigious speech performed for the opening of REAACH-ID symposium 


Sciences and Philosophy. 
During the symposium the scholars asse 


of Artificial and Virtual Reality offers a ne 


thanks to effects created by the digital too 
observer as a real experience. 


tions, underlining the links between Al and 


that is concrete and material, therefore something natura 
w version of reality in itself. lt cannot be consid- 
ered anymore as pure vision, but as a participation of all the senses, even of the whole body, 


Is which are so li 


indicated how Artificial Intelligence may show new possibilities in the world of representa- 
Engineering, Information Technology, Cognitive 


rted several times that reality means everything 


and changeable, but the advent 


kely that they are accepted by an 


It has been noted that artificial reality is the most advanced form of interaction between 
man and machine. Inside this deceptive reality everything that is perceived is generated by 
a computer that responds to our movements with images and sounds designed to give us 


+ 


towards a new kind of utopia. 


+ 


to digital technologies that increase their reading and interpretation in the process of | 
analysis, design and enhancement. For over fifteen years, the Italian teachers of Drawing 
have already made use of a fruitful intertwining between Artificial In 
ed Reality thanks to the new possibilities of identification and connection between digital 
products and physical consistencies, in a mix of real and virtual world. 
The way of acting, according to the methodological profile of our research, which start: 
from physical space, has found in the digital world and Artificial Intelligence those tools 


the illusion of a virtual world that breaks the laws of physics projecting our self in a free 


+ 


space-time. Artificial Reality represents the realization of an invention, a trespassing tool 


f it is true that Artificial Reality involves all the senses with illusory messages, nevertheless 
visual messages have the upper hand over all the others; it is precisely for 
during the symposium the scholars have exalted the world of Artificial Reality as a real op- 
portunity for exploring and visual communicating. The user, wandering without constraint 
into a new world, becomes an experimenter of new models of thought and technologies 
Furthermore, researchers in our disciplinary sector often study cultural heritage resor 


+ 


this reason that 


ts 


ting 


+ 


US 


+ + 


telligence and Augment- 


+ 


S 


for expanding the reality (for this reason called ‘“augmented’) aiming at redefining the 


+ 


way to share cultural heritage, or the way to enhance it through innovative systems of 


community participation. 


The success of REAACH-ID encouraged Roberta Spallone, Andrea Giordano and Michele 
Russo to organize a second symposium. For this | feel the need to thank them for the good 
work produced in this volume, which collects the results of the open discussion and the 
scholars’ research presented during the first symposium, as well as for what they are pre- 


paring in the next future meeting. 
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Representation Challenges: 
The Reasons of the Research 


Andrea Giordano 
Michele Russo 
Roberta Spallone 


Augmented Reality (AR) and Artificial Intelligence (Al) are technological domains that 
closely interact with space at architectural and urban scale in the broader ambits of 
cultural heritage and innovative design. The growing interest is perceivable in many fields 
of knowledge, supported by the rapid development and advancement of theory and 
application, software and devices, fueling a pervasive phenomenon within our daily lives. 
These technologies demonstrate to be best exploited when their application and other 
information and communication technology (ICT) advancements achieve a continuum. In 
particular AR defines an alternative path to observe, analyze and communicate space and 
artifacts. Besides, Al opens future scenarios in data processing, redefining the relationship 
between man and computer. 

n the last few years, the AR/Al expansion and relationship have raised deep trans-disci- 
plinary speculation. The research experiences have shown many cross-relations in Archi- 
tecture and Design domains. Representation studies could arise an international debate 
as a convergence place of multidisciplinary theoretical and applicative contributions relat- 
ed to architecture, city, environment, tangible and intangible Cultural Heritage. 

A research synergy between different Universities (Politecnico di Torino, University of 
Padua, Sapienza University of Rome) was the first occasion to share collected experienc- 
es. The discussion between scholars led to the importance of fixing a first state of the 
art about AR/AI topics in the national Representation area, analyzing the role of AR/AI in 
the chain of knowledge. Besides, critical interest has been focused on exploring the disci- 
plinary boundaries, identifying affine disciplines involved in similar research, and activating 
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Fig. |. Snapshot of the 
Zoom platform during 
the conference. 
Editing: M. Russo. 


effective relationships. Working beyond the usual research edges meant evaluating wheth- 
er our discipline could expand tts field of action, establishing as challenge new directions 
of research, possible partners, priorities, and objectives. 

Our initiative started in the fall of 2019, with a series of international events related to 
research works based on possible intersections between AR and Al and the world of 
Representation. The importance of opening a discussion with our colleagues at national 
level was confirmed during these events, which were supported by Francesca Fatta [1], 
who advocated a meeting initiative. So, we created a first working group, formalized 
in the Scientific Committee, which includes the promoters: Francesca Fatta, Salvatore 
Barba, Marco Giorgio Bevilacqua, Stefano Brusaporci, Alessandro Luigini, Cettina San- 
tagati and Alberto Sdegno.This group discusses the possibility to organize a Symposium 
intended as a national meeting framed in the Representation field, which took place on- 
line on 13 and 14 October 2020, managed by the Zoom platform of the University of 
Padua (fig. 1). The call for this virtual event met the interest of |75 authors, mostly com- 
ing from national Universities, proposing 83 research experiences. The proposed topics 
were divided into three main areas of interest: Augmented Reality, Artificial Intelligence, 
Virtual Reality & Digital Modeling. The related abstracts were reviewed and divided 
into 44 oral and 39 video presentations. This division wanted to preserve the duration 
of the event (two days) giving to everyone the opportunity to present their research. 
The video presentations were uploaded before the beginning of the conference on 
a dedicated YouTube channel and shared with all Symposium participants. In addition, 
the final recordings of the event were uploaded on the same channel, ensuring that all 
contributions could be reviewed. The Symposium counted 265 scholars during the two 
days (fig. 2). The meeting days were structured with plenary communications, proposing 
eight thematic sessions. After a brief introduction by the UID President (Francesca Fat- 
ta) and by the three organizers, each day featured four sessions, opened by a keynote 
speaker. The seven invited speakers belonging to different disciplinary fields showed 
the potential of AR and Al in their specific discipline, highlighting, on the one hand, 
the possible connections with the field of Representation while confirming the value 
of the transdisciplinary nature of our studies. The invited speakers were Mario Rasetti 
(ISI Foundation), Claudio Casetti (Politecnico di Torino), Michele Bonino (Politecnico di 
Torino), Simone Milani (University of Padua), Fabrizio Lamberti (Politecnico di Torino), 
Eleonora Grilli (Fondazione Bruno Kessler — FBK), Alberto Tono (Computational Design 
Studio — Studio HOK). 


Federico Mario L... 


Barbara Aterini 


Fig. 2. Spatial diagram 
showing the distribution 
of research scholars ac- 


cepted at the Symposium. 


Editing: M. Russo. 


The volume that we are printing today follows the steps of the Symposium: the discussion 
that emerged from it has fed the proposal of the published essays, which have been subject- 
ed to peer review. In addition, the request to prepare a research paper after the meeting 
gave the authors the possibility to frame their research experience in an aware context, 
suggesting cross-references with other colleagues. The themes developed were collected in 
eight lines described below, which may guide future research developments. 


AR&AI Theoretical Concepts 


While augmented reality has been part of the disciplinary interests of Representation for 
at least twenty years, the much more recent developments in artificial intelligence and 
the possibilities of connecting the potential of the two technologies triggered significant 
theoretical reflections. The contributions in this section of the book range from the his- 
torical and cultural background, to the possible challenges of Representation, opening to 
other areas of knowledge. 

An historical excursus has been traced by Alberto Sdegno who brings to attention the 
concept of ‘Drawing Automata’ that he discusses in the sense of automata that draw. The 
author crosses some singular experiences like those with ‘mechanical puppets’, up to de- 
velopments in graphic technology for digital drawing, since the first virtual reality systems 
and the use of artificial intelligence aimed at shape recognition. Finally, he affirms that, in his 
opinion, the most interesting experiments will be the ones carried out in the field of shape 
grammars, and in particular those in which the role of architecture is involved. 

The link with disciplinary roots is explored by Francesco Bergamo who tries to answer 
one of the core issues of the discussion: the role of Drawing in representing and designing 
new epistemological models during the current Al spread, especially in relation to today's 
available advanced tools. He observes that Drawing always played a major role in mod- 
eling knowledge and explains that the technical achievements, deployed while facing the 
current challenges, should be connected with the historical and critical background pro- 


vided by the genealogies of perspective and maps. On the same line, Marco Vitali, Giulia 
Bertola, Fabrizio Natta & Francesca Ronco note that the links between Representation 
and artificial intelligence involve many fields of architectural research, recording contin- 
uous and significant advances that require a constant update of the state of the art and 
a careful consideration of the role of Representation in interdisciplinary researches. For 
this reason, they investigate the most cutting-edge case studies in the fields of Cultural 
Heritage, Museums, and Digital fabrication. 

Then, Giorgio Buratti, Sara Conte & Michela Rossi explore the new frontiers of Al ap- 
plied to Cultural Heritage and identify three main lines of research in this field that exploit 
the synergy between machine learning, big data and Al. They deal with: the development 
of innovative methodologies and systems to monitor and prevent damage to the cultural 
heritage; the experiments that focus on automatic data acquisition, such as the application 
of mobile technologies for three—dimensional surveying and acquisition of CH images; the 
management of large amounts of analog or digital data and information. Also Valerio Palma 
notices the unprecedented opportunities of advances in information and communication 
technologies for the documentation and analysis of architectural and urban assets. Nev- 
ertheless, he warns of the risk that the schematic understanding of reality produced by 
eidomatic tools, i.e. automatic recognition and classification become a black box, that is, 
algorithms we accept the results without questioning how these are produced or whether 
they can be adapted to changing needs. Then, Palma summarizes three connected research 
experiences he has been part of. They produced apps and prototypes exploiting geospatial 
databases, Al for image recognition, and AR based on model tracking, applied to architectur- 
al heritage, contributing an insight on the interpretation of the built environment form as a 
means to produce shared knowledge and operational outcomes. 

Other authors develop broader reasoning regarding the design and ethical impact of new 
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technologies. It is the case of Claudio Marchese & Antonino Nastasi that observe, in the 
digital age, the persistence of time and space as coordinates for human presence and their 
intrinsic relationship with Architecture. They analyze as sort of ‘scenarios’ concerning pre- 
figuration, measurements, and relationships, dealing with Al and AR combinations in a pro- 
jectual perspective. Moreover, Marco Ferrari & Lodovica Valetti analyze the ethical issues 
triggered by the integration of new technologies in the management of Cultural Heritage, 
in particular historical sites and museums, related to ontological, economic, and social as- 
pects, trying to outline some principles able to limit these critical implications. They focus on 
the authenticity of reproduction, the costs of VR and AR systems, and individualism due to 
the use of personal devices. The scholars conclude that the relationship between man and 
machine and the concepts of real and virtual must be kept at the center of innovation and 


application in this field. 


AR&AI Virtual Reconstruction 
Digital convergence in the last decades has favored the development of virtual reconstruc- 
tions for heritage enhancement. These are connected with documentary analysis and digital 
survey, dialoguing both with textual, iconographic and archival sources, and with artistic, 
archaeological, architectural and urban consistencies. Artificial intelligence and augmented 
reality are intertwined with knowledge, interpretation and communication, enhancing the 
value of virtual reconstructions from the point of view of scientific validation, knowledge 
construction, and inclusion. 

The dialogue with architectural literature is interwoven by Alessio Bortot, who develops 
his reasoning connecting the treatise of stereotomy by Monduit with AR technology with 
the aim to better describe the tracing of stone cutting. Starting from the interest for de La 
Rue stereoscopic approach, which gives the opportunity to go beyond the two-dimen- 
sional limits of paper surfaces, such as the use of pop-ups, the scholar proposes the use of 
AR for creating digital pop—ups. The reconstructive 3D models of the treatise plates were 
provided with different semantic values and, through AR, could be visualized and split into 
blocks, while the relationship between form and structure was shown through the use 


12 


of a false color Fem analysis. Similarly, Cristina Candito, Andrea Quartara & Alessandro 
Meloni study the scientific sources to deal with the representation of six Keplerian stel- 
ated dodecahedrons in the decoration of an aulic room in an historical palace in Genoa. 
Through the virtual reconstruction of the illusory space frescoed on the vault, they illus- 
trate in a communicative, but not simplified way, through an immersive AR experience, 
the connections between the Genoese decoration and the wider seventeenth-century 
scientific milieu. 
The analysis of archival and bibliographical sources, coupled with digital survey, supports 
the considerations of Maria Grazia Cianci, Daniele Calisi, Sara Colaceci & Francesca 
Paola Mondelli in light of the assignment by the municipality of Rome for the realization 
of three competition dossiers. They reflect on how and to what extent the Public Ad- 
ministration can now benefit from new technologies and digital tools, and conclude that 
although the current huge step is towards the digitization of the architectural heritage, 
it is also desirable to proceed towards Virtual Reality to ensure immersive design for 
competitors. Also Maurizio Marco Bocconcino & Mariapaola Vozzola present remarks 
that focus on analysis at urban scale, in order to propose, in this case, solutions aimed at 
safeguarding memory. They show the methodological framework for a research project 
on conservation and care of the memory of changing urban contexts and highlight the 
need of implementation of integrated digital archives with interactive and hypertextual 
navigation between the assets themselves. 

Virtual reconstructions, linked to AR and Al technologies, are particularly developed 
in the archaeological field. It is the case of the research by Riccardo Florio, Raffaele 
Catuogno, Teresa Della Corte & Veronica Marino. They set up a methodology to 
deepen the investigation on the current consistencies of the Roman bath of Cumae that, 
through the preparation of a suitable neural network managed by Al processes, are able 
to formulate reconstructive hypotheses with scientific authority. Moreover, through the 
adoption of web-based and open source technologies and the applications of AR and 
interactive digital visualization, multiple information are associated with the 3D model. 
Also Maurizio Perticarini & Chiara Callegaro deal with archaeological heritage and pro- 
pose the creation of an adaptive AR app with the aim to enhance through cultural tour 
ism the vestiges of ancient buildings hidden by urban building development of Padua. The 
workflow identifies a path through the city and connects the analyzed and represented 
artifacts through documents, digital survey, BIM modeling and, finally, AR app, foreseeing 
further insights through artificial intelligence to differentiate the contents according to 
the user's preferences. 

Finally Antonio Calandriello exploits ICTs, i.e. digital reconstruction, virtual and augment- 
ed reality, video production, and application development, aimed at the enhancement of 
a Renaissance palace in Padua. In this way, it is possible to increase the inclusiveness and 
raise the level of use, giving visitors the opportunity to virtually visit the building and its 
precious frescoed halls, which the public cannot visit. 


AR&AI Heritage Routes 


Built and environmental heritage are very often made up of a series of artefacts and el- 
ements that take on a particular value in their constitution as a system.The construction 
and use of heritage routes can profitably avail of augmented reality and artificial intelligence 
technologies for the construction of knowledge and its sharing as well as for flow control 
and route recommendation.The researches presented in this section demonstrate the de- 
sire to create closer links between the real and the virtual sets, realizing a real continuum 
between them. 

n this sense moves the work of Stefano Brusaporci, Pamela Maiezza, Alessandra Tata, 
Fabio Graziosi & Fabio Franchi dealing with the idea that a smart heritage could be realized 
through VR/AR/MR combined with Al applications. Their observations are based on a wide 
experience on the territory of L'Aquila that highlights the potential of the so—called ‘phygital’ 
with the protection and enhancement of assets often characterized by important elements 
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of fragility. In the same direction, although referring to environmental heritage, Alessandra 
Pagliano, Greta Attademo & Anna Lisa Pecora propose a systematization of the knowledge 
of the Phlegraean Fields Park, through surveys and 3D models, integrated by the use of differ- 
ent digital technologies, AR in particular Each site becomes the node of a network of thematic 
routes, traced starting from the major attractions of the area. The goal is to define a hybrid 
landscape, made of site specific visits and digital experiences, that generate a new model of 
inclusive museum, configuring cultural relationships between physically distant places. Also An- 
drea Rolando, Domenico D’Uva & Alessandro Scandiffio work on environmental heritage 
enhancement. They aim at investigating the potential of image segmentation technology, based 
on web application, for measuring the spatial quality of slow routes. The method has been ap- 
plied to some stretches of slow—mobility routes that are localized along a fragile coastal land- 
scape of southern Italy. Such an update of the tool could take advantage of Machine Learning 
systems, namely the ability to learn through analysis, to improve its precision and reliability. 
The research of Gerardo Maria Cennamo is engaged in the so called ‘open air museums’ 
inking in a single path a series of cultural stops. The author refers to two ongoing experi- 
ences in Naples and Rome where, thanks to GPS systems it is possible to develop dynamic 
paths of perception of the archaeological and architectural heritage, integrated by augment- 
ed reality applications that realize an active involvement of the visitor through pedestrian 
and vehicular itineraries. 

Educational aims address the cutting-edge research by Alessandro Luigini, Stefano Brusapor- 
ci, Alessandro Basso & Pamela Maiezza that present a project addressed to the documenta- 
tion, investigation of architectural values, and valorization of a Sanctuary in Pescara, through an 
application of augmented reality technology, able to visualize the previous configuration of the 
church, enhanced by an artificial intelligence—based tracking application. The awareness-raising 
program includes online communication and activities with local schools.Also Serena Fumero 
& Benedetta Frezzotti describe a didactic experience related to the use of AR in the pro- 
motion of heritage sites. The low cost and the relative ease of creating content of the cultural 
experience will encourage a similar approach in many other heritage sites. 
The heritage route set up by Marinella Arena & Gianluca Lax develops a strategy to 
guide users and scholars in the Byzantine iconographic world, highlighting the elements that 
contribute to recognizing the sacred figures represented. The artificial-intelligence—based 
technique used to analyze 19 images of Saint Nicholas set up some preliminary strategies 
for the morphological classification of the iconography through the identification of the 
proportional relationships between the parts of the face. 

n an original way, Ornella Zerlenga, Vincenzo Cirillo, Massimiliano Masullo, Aniello 
Pascale & Luigi Maffei create a philological heritage route investigating the ephemeral 
architecture of the eighteenth century in Naples. The goal is an unprecedented represen- 
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tation, modeling and fruition project, accompanied by a graphic visualization of the places 
to better understand the architectural and urban space. An Augmented-Reality application 
prototype makes it possible to disseminate the compositional elements of the celebrations 
to a non-specialist public. 
Finally, Giorgio Verdiani, Ylenia Ricci, Andrea Pasquali & Stephane Giraudeau set up a 
real route between different research experiences personally carried out, that links to- 
gether digital modelling of no longer and still existing architectures with AR, VR and XR 
experiences. Not only in the will of capturing the users, but in the true belief that Digital 
Heritage is an occasion allowing an easier reading and an option for spreading the knowl- 


edge of Cultural Heritage. 


AR&AI Classification and 3D Analysis 


AR & Al classification and 3D analysis is an increasingly important area of research through 
data segmentation and attribution of specific connotations, supporting the architectural 
analysis process. In particular, the research focus shifts from 3D acquisition to data pro- 
cessing, defining hierarchical-oriented information systems. This strategy implements the 
geometrical data structures for reality-based and interpretative definition of model, opti- 
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mizing the model communication passage through AR/MR/VR systems. The papers cover 
the entire process, from the treatment of the survey data to the Scan-to-BIM passage, up 
to the building's analysis and digital heritage enhancement. 

The first group of articles is oriented towards photogrammetric techniques underlying data 
processing. Marco Limongiello & Lucas Matias Gujski work directly on data acquired by 
UAV imagery. They suggest an innovative approach that can support survey methods by 
applying Al algorithms, improving the accuracy of extracted point clouds. Some parameters 
are analyzed through a Self-Organizing Map (SOM) to reach a compromise between the 
variable values, the noise reduction, and the 3D model definition, allowing a fast data clus- 
tering. Similarly, Andrea Tomalini, Edoardo Pristeri & Letizia Bergamasco start from the 
UAV imaging approach and suggest the use of a photogrammetric model to generate new 
image datasets for training Machine Learning (ML) algorithms and recognize built heritage. 
They apply Physically Based Rendering (PBR) tools to extend the image datasets with artifi- 
cial data. In the end, Pierpaolo D’Agostino & Federico Minelli propose a methodology for 
the automated delineation of brick masonries from images to a vector representation. The 
edge detection and vector delineation of brick joints allow a brick clustering for masonry 
classification, implemented in AR visualization. 

The second group is oriented to the segmentation of point clouds acquired by 3D scanners 
for data classification with the aim to study architecture. Michele Russo, Eleonora Grilli, Fabio 
Remondino, Simone Teruggi & Francesco Fassi present an iterative multi-level and multi- 
resolution classification process that improves the learning activity based on a supervised au- 
tomatic Machine Learning approach to optimize 3D point cloud classification by a hierarchical 
concept. This strategy requires few annotated 3D data while very detailed semantic segmen- 
tation results can be achieved. Paolo Clini, Roberto Pierdicca, Ramona Quattrini, Emanuele 
Frontoni & Romina Nespeca suggest an innovative approach based on HBIM existing models 
to improve the segmentation of 3D point clouds through the Deep Learning approach. The 
proposed methodology facilitates Scan-to-HBIM passage, generating a sufficiently large data 
set of synthetic clouds to train the neural network. The last paper referred to this cluster is a 
work by Federica Maietti, Marco Medici & Ernesto ladanza and focuses on Al and 3D point 
cloud models, showing a new integration within the BIM environment, image classification 
algorithms, point cloud segmentation, and representation models. This paper presents a pre- 
iminary experience focused on the Al for processing the large amount of data obtained from 
digitization processes applied to the Architectural Heritage. 

n the last three research papers, the modeling activity based on acquired geometrical 
data assumes a strategic value. In such a sense, Elisabetta Caterina Giovannini presents an 
extended analysis of the current methodologies and workflows for data modeling in Archi- 
tecture, deepening the use of standards, descriptive metadata, and ontologies for Cultural 
Heritage representation. Furthermore, this research underlines the complexity of creating 
conceptual models able to manage three-dimensional data and descriptive metadata on 
architectural heritage documentation. 
The last two articles share common themes of museums and IT, though explored from two 
different points of view. Massimiliano Campi, Valeria Cera, Francesco Cutugno, Anton- 
ella di Luggo, Domenico lovane & Antonio Origlia propose a methodology of collecting, 
analyzing, and modeling multimodal data, helpful in designing virtual agents in 3D museum 
environments. This research is part of the national CHROME project, which deepens the 
interconnections between representation, survey, and information technology domains. The 
architecture focuses on graph databases and 3D models, using modern peripheral devices 
and third-party services for capturing and analyzing input signals; it also integrates prob- 
abilistic decision-making systems for controlling interaction in 3D environments. Besides, 
Marco Giorgio Bevilacqua, Anthony Fedeli, Federico Capriuoli, Antonella Gioli, Cosimo 
Monteleone & Andrea Piemonte focus the attention mainly on the communication muse- 
um strategy. They suggest a new visit path based on immersive experiences of video map- 
ping, VR/AR, sound immersion, informative totems, audio—visual supports, and multisensory 
activities. The project aims to investigate the potential in applying new AR/VR technologies 
to enhance virtual tours and interpret 3D models of real environments. 
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AR&AI Urban Enhancement 


This section is dedicated to research that refers mainly to the urban space and the territorial 
scale, considering also building knowledge connected to the territory and the architecture cul- 
ture. The central topic of this section is the cultural and sensory itinerary, allowing experiences 
that go beyond the visible. The use of immersive and augmented virtual reality techniques 
creates a link between the tangible and intangible, memory and existing real space, underlying 
the construction logic, the building itineraries spreading knowledge. Both urban systems and 
single buildings are spatial and historical landmarks. Their disappearance or damage may erase 
the relation between present and past, causing a sense of disorientation. The first articles ex- 
plore AR/VR methodologies for the memory recovery of places destroyed by earthquakes. 
Giuseppe Amoruso, Polina Mironenko & Valentina Demarchi propose a path of memory 
re-appropriation through multimedia installations, emphasizing the evocative interaction be- 
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tween reality, memory, and reproduction of the intangible to promote collective memory. 
Thus, the real environment becomes a temporary and experiential museum, expanding the 
immaterial dimension of accessibility discovery. Besides, Caterina Palestini & Alessandro Bas- 
so suggest a solution to promote the use of the historical buildings, activating at the same 
time the dynamics of cultural regeneration in the area, using AR+AI systems for a tour that 
can show the urban architectural evolution, monitoring and enhancing the knowledge and the 
transformations undergone by the architectural heritage. 
Concerning urban routes, the experiential factor plays a central role. The experience occurs 
mainly through the human vision, which is dominant in AR/MR/VR development without ex- 
cluding other senses. Alessandra Pagliano presents a project aiming at preserving the memory 
of murals using AR. In this way, the narrative message of the old murals is expanded thanks 
to the digital multimedia contents, reinforcing the storytelling, the cultural connection, and the 
memory of the place. On the other hand, Fabio Bianconi, Marco Filippucci & Marco Secca- 
roni suggest using EEG helmets synchronized to GPS to collect human cognitive conditions 
and transform them into representations that reflect the people's feelings during their itinerary. 
This research aims to define methodologies based on interdisciplinary interpretative criteria 
to understand the human-environment impact, directing design choices to improve wellbeing. 
As mentioned earlier large architectural spaces can be compared with those at urban 
scale, especially if the aim is to communicate and promote them through a more profound 
knowledge. This is the case of Marco Canciani, Giovanna Spadafora, Mauro Saccone & 
Antonio Camassa, who present two AR applications about important architectonic space, 
with the final aim to develop some AR multimedia content linked to semantic concepts. The 
proposed experiences use AR to share the analyses carried out on drawings and canvas, 
unveiling the constructive meaning of space. 

The last cluster is dedicated to communication and promotion of local assets and retails, 
always framed through knowledge itineraries. Paolo Belardi, Valeria Menchetelli, Giovanna 
Ramaccini, Margherita Maria Ristori & Camilla Sorignani present the concept of “Aug- 
mented Retail,” representing a strategy to achieve an Augmented Identity, changing the 
retail space from a ‘product window’ to an ‘experiential theatre’. This research aims to open 
new perspectives based on AR and Al to enhance local identity promoting itineraries of 
historic shops. Regarding local assets, Daniele Rossi & Federico O. Oppedisano, illustrates 
the potential of AR in the valorization of food and wine heritage. They highlight the playful 
interactive capacities of this technology shifting towards renewed communication models 
that aim to extend the cultural offer in an increasingly rapid and immediate form. 


AR&AI Museum Heritage 


In the digital era, museums have seen a significant expansion of spaces and tools for the dis- 
semination of knowledge. In this sense, the digital takes on multiple roles precisely articulating 
itself between Al and AR for new ways of disseminating and communicating Cultural Heri- 
tage. In this sense, Massimo Barilla & Daniele Colistra are involved in the creation of an im- 
mersive and interactive room — Nello Scill'e Cariddi — in which the marine environment of the 
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Strait of Messina is used as the first experimental scenarios to develop and test technology 
AI/AR.The room, therefore, transforms film collections, integrating them with the production 
of specific images, into virtual environments containing structured catalogs and into interac- 
tive installations for educational, playful, scientific and popular use. Francesco Borella, Isabella 
Friso, Ludovica Galeazzo, Cosimo Monteleone & Elena Svalduz then deal with the cultural 
interfaces that make a video installation which is also immersive and interactive. Their aim is 
to keep a high and active visitor's attention, in the video dedicated to the Academia insula 
of Venice several short-term sections have been introduced, which can be interrogated and 
activated by sensors, while a Sound Shower system facilitates the understanding of the mul- 
timedia story by listening to the voice of a storyteller The contribution of Laura Carlevaris, 
Marco Fasolo & Flavia Camagni is also fundamental, dealing with the options provided by 
the recent applications of Augmented Reality (AR) to the knowledge and enhancement of a 
cultural asset: the perspective in wood inlays. The intention is therefore to use perspective in 
AR ambient as the ideal tool to virtually experience the space represented in the exhibited 
decoration. The creation of an experiential path linked to perspective, organized by a sensi- 
tive narrative that begins in the very act of its use, to allow the observer a critical interaction 
dependent on the process of knowledge of the works and their evoked spaces. This is what 
Gabriella Liva & Massimiliano Ciammaichella propose for the perspective analysis of some 
canvases by Jacopo Robusti. Furthermore, for the enhancement of Cultural Heritage it is 
important to explore the potential of the combined use of Augmented Reality (AR) and 
rapid prototyping. Giuseppe D’Acunto explores how these innovative methodologies can 
be effective in the creation of museum installations able not only to show the contents of 
an exhibition in an original and attractive way, but also to recover the memory of a place 
by reconstructing the original position of the environment, as in the case of the installation 
of Palazzo Grimani, where the sculptures have moved from their original position over the 
centuries. These aspects linked to the relationship between AR and VR for the installations 
are addressed by Giuseppe Di Gregorio, who highlights the workflow produced for the 
realization of the 3D models of one of the three churches in the rocks of the necropolis of 
Pantalica, an Unesco site. In this sense, the ongoing process to achieve the realization of VR 
and AR models to use with three different levels of immersive reality is important: normal 
commercial viewers, different types of Oculus viewers, or in special virtual rooms Cave 
Automatic Virtual Environment (CAVE). In this sense, the paper of Elena Ippoliti also deals 
with the research experiences of the last 15 years in which the potential of digital technology, 
and of Augmented Reality in particular has been tested to enhance Cultural Heritage. This 
research outlines the value of preliminary digital simulation for form-finding, both in terms 
of the actual final configuration of the project and in terms of the less tangible aspects of the 
efficiency of building. As a result, representation takes on a new position as the ‘place’ of the 
model. The dynamic passage between real and virtual in a spatial model helps in representing 
intangibly, with high reliability, what is concretely abstract. The challenge of the use of Cultural 
Heritage by subjects with sensory limits has also become fundamental for museum instal- 
lations. Franco Prampolini, Dina Porpiglia & Antonio Gambino propose the 3D survey of 
all the main finds to be exhibited with analytical photo—modeling techniques, their scientific 
cataloging, the creation of a website with a high interactivity content and an application that 
allows sharing extended information with blind people, through the combined use of analog 
3D models and AR creation software. The topic of facial recognition technologies — already 
used today in many applications, to support security systems in sensitive buildings — is anoth- 
er developing theme for museums or art galleries. Indeed, Paola Puma & Giuseppe Nicastro 
describe an experimentation in a museum of a research in order to provide a tool capable 
of interpreting the reaction of a user in front of a work of art, proposing a responsive infor 
mation content with what is manifested, through facial expressions. Finally, it is important to 
highlight how generative design is now fundamental for the creation of interactive digital con- 
figurations, aimed at the creation of artistic languages and for the representation of dynamic 
contexts. In this sense, Leopoldo Repola, Nicola Scotto di Carlo, Andrea Maioli & Matteo 
Martignoni have integrated a narrative process, linking aspects of recording the movement 
of bodies in water with the principles of data visualization. 
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AR&AI Building Information Modeling and Monitoring 


AR & Al are currently playing an important role in the advanced implementation of BIM 
with special reference also to the monitoring of architectural and engineering artifacts. And 
there are many applications also in terms of Scan-to-BIM, such as the essay of Vincen- 
zo Bagnolo, Raffaele Argiolas & Nicola Paba which present the first results of an ongo- 
ing research on the advantages of implementing computational modeling in Scan-to-BIM 
processes for the representation of historical architecture in AR and VR applications for 
educational and communication purposes. But if we consider that the conservation and 
regeneration of the existing built heritage is still characterized by an inefficient manage- 
ment of time and costs, throughout the life cycle, it is central the research proposed by 
Marcello Balzani, Fabiana Raco & Manlio Montuori which focus on the development of 
integrated digital solutions for the acquisition, modeling and visualization of data relating to 
the building and construction supply chain. At the same time Fabrizio Banfi highlights the 
convergence of innovative methods, latest generation of technologies and software appli- 
cations for the representation, archiving, transmission of the material and immaterial values 
of architecture; for these reasons, he researches the development of a cloud—based open 
source BIM platform and XR/AI projects able of sharing a knowledge process based on 
new levels of interactivity and digital creativity. Carlo Biagini, Ylenia Ricci & Irene Villoresi 
also highlight how, in recent years, the application of BIM to Cultural Heritage has led to 
the development of solid operating methods that have enabled more efficient information 
management. They then tested whether BIM models can be exploited to create immersive 
experiences in digitally simulated environments, setting new ways of viewing and evaluating 
the built space. In this sense it is important to test the accuracy of the solutions adopted 
using BIM models, in a representation of the built structure that is also able to summarize 
the qualities to be detected. Fabio Bianconi, Marco Filippucci & Giulia Pelliccia, with their 
research, have developed some case studies in the field of wooden constructions, inserting 
themselves in a framework that emphasizes the relationship between simulation and reali- 
zation. The introduction of automation methodologies through the use of deep learning of 
BIM modeling starting from different types of formats, such as digital processing of paper 
documents and CAD formats, is also important for the relationship between BIM and Al. In 
this sense, David Campagnolo & Paolo Borin emphasize as a proof of concept of a possible 
contribution that a technique currently barely adopted in the architectural field — such as 
deep learning — can lead to the design, in particular in the creation of the information model, 
an activity that today takes time. If we then consider that in the digital age the construction 
industry has seen significant changes in the design, construction and learning of spatial pro- 
cesses through new technological systems, it is important to highlight how these processes 
influence the management and the way in which data are collected, cataloged and moni- 
tored using sensors and connected users. Matteo Del Giudice, Daniela De Luca & Anna 
Osello therefore define new safe and resilient digital models combined with interoperable 
methods that minimize the impacts of our built heritage during its life cycle. In this direction 
we find the essay of Raissa Garozzo who proposes a new methodological approach for 
the evaluation of the health status of railway bridges in masonry based on the definition 
of image-based and Al-driven investigation protocols useful for the creation of semiauto- 
matic H-BIM models. Similarly, the research of Federico Mario La Russa which provides an 
overview of the evolution of the VPL and an application case concerning the classification 
of seismic vulnerability indices with Al. An important topic involved in this volume regards 
also energy monitoring that can be implemented in an advanced way with BIM as in the 
research of Marco Filippucci, Fabio Bianconi & Michela Meschini who analyze, in particular, 
those buildings that, even if of different eras and construction technologies, require energy 
requalification. Finally, the contribution of Assunta Pelliccio & Marco Saccucci proposes a 
methodology (DMM.R,., acronym for Design, Virtualization, Modeling, Reproduction), which 
in four temporally consequential phases builds a tool able of providing territorial, envi- 
ronmental, architectural and historical information of a particular case study (an industrial 
building) but extendable to the built heritage. 
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AR&AI Education and Shape Representation 


A particular area of development of AR & Al could be certainly linked to educational pur- 
poses. Indeed, there are many repercussions that we can apply to students’ training especial- 
ly for aspects concerning inclusion and support. For example, the difficulties involving clinical 
autistics are mainly related to the perception deficit, therefore VR/AR & Al can become a 
valid support for people with ASD, improving relationships with space and people. Anna 
Lisa Pecora, in this sense, tries to provide a guiding tool for a human-centered VR design. 
So, at an academic level of education, AR & Al may facilitate the preparation in faculties such 
as Architecture and Engineering. For example, Emanuela Lanzara & Mara Capone propose 
how to improve a dataset of generative algorithmic definitions capable of returning an opti- 
mized “semi-ideal”’ curve that best fits a generic profile based on reality, starting from some 
of tts points, with interesting effects in the field of education. Therefore, learning through 
the direct experimentation of “intelligent” models — in their variety of manifestations and 
hybridizations — is undoubtedly a very powerful aid in the acquisition of knowledge. This is 
what Marta Salvatore, Leonardo Baglioni, Graziano Mario Valenti & Alessandro Mar- 
tinelli assert, entering into the specific role of the architectural configuration: testing or 
experimenting with models helps not only to understand the shape of existing architecture, 
but even more to imagine and design new buildings. In this sense, the essay proposed by 
Roberta Spallone & Valerio Palma presents methodologies, objectives and some of the re- 
sults of a university course — Techniques of Digital Representation — in which the interaction 
between the different digital representation techniques in relation to AR and Al technolo- 
gies is developed, providing tools for investigating critical aspects. Finally, it is import 


tant to 
underline the repercussions that training in these terms has for architecture, engineering 
and construction (AEC) industries: Alberto Tono, Meher Shashwat Nigam, Stasya Fedor- 
ova, Amirhossein Ahmadnia & Cecilia Bolognesi test algorithms for 3D reconstruction 
from a single image, specifically for building envelopes, with end-to-end geometric deep 
learning approaches. At the same time, the importance of this kind of apprenticeship can 
be recognized in the touristic industry, from a communication point of view, as Maria Linda 
Falcidieno, Maria Elisabetta Ruggiero & Ruggiero Torti affirm: their research introduces 
some insights and proposals shared with the operators of the sector and aimed at the in- 
telligent reformulation of approaches and languages in order to bring potential customers 
closer to the Cruise Experience. 


Notes 
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scholars of Representation. 
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AR&AI 
theoretical concepts 


REPRESENTATION CHALLENGES 
Augmented Reality and Artificial Intelligence in 
Cultural Heritage and Innovative Design Domain 


The Role of Drawing in Data Analysis 
and Data Representation 


Francesco Bergamo 


Abstract 


This paper discusses the role of Drawing in representing and designing new epistemological models 
during the current Al spread, especially in relation to new advanced tools available to scientists. The 
accelerated development of technologies for data visualization and immersive and augmented expe- 
riences consolidates shared workflows, while criticism and genealogies of such tools are too often 
left aside. Therefore, scholars have not only to develop new technological tools and applied meth- 
odologies but also to work on shared sets of theoretical concepts that are necessary foundations to 
designed contents. The history of representation — especially maps on one side, perspective on the 
other — provides most of the necessary common ground, which is much needed if we choose that it 
is still worth governing Als as much as we can, instead of abandoning ourselves to ‘the end of theory’, 
which would imply the end of any meaningful scientific drawing made by and for humans. 


Keywords 
augmented reality, artificial intelligence, DataViz, heritage, interface design. 
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“It's easy to forget that a lot of what we do as scientists comprises choosing how to rep- 
resent our work. These choices are made not only for larger, public-facing events but also 
among ourselves at individual meetings, small conferences, and even in our emails. Osten- 
sibly, in parallel, there's also the question of how much agency the scientist-qua-artist has 
over the work and its import” [I]. 


Immersive vs. Disembodied Representations 


The many speakers and participants in the REAACH-ID Symposium showed several ad- 
vanced applications of Augmented Reality and Artificial Intelligence in scientific research, 
especially in the fields of drawing and representation for cultural heritage and design, 
besides interdisciplinary exchanges aimed at building innovative tools for analyzing data 
and augmenting the user's experience while exploring information — be it for promoting 
or studying heritage, design approaches, etc. 
A first rough classification of the theories and practices of representation, in relation to 
the new modes of production and communication of knowledge, would split common 
contemporary representational artifacts into two types, apparently antithetical. On the 
one side, immersive applications and interfaces, where perspective goes ‘hyper’ by taking 
the observer out of the frame of Alberti's window and into a virtual or augmented world. 
On the other, charts, maps, and diagrams, often looking much like traditional representa- 
tions of networks: they are more and more non—perspectival, abstract, looking much like 
each other their form more and more independent from that of the represented object, 
which often does not have a proper form itself (e.g., what does a pandemic ‘look like’?). 
The scientific bases of both types in the contemporary world can be found at the origins 
of the Modern era: the first coming from the western perspectival tradition [2] and the 
second from cartography but also from the many attempts to represent abstract theories 
or ideas — and their underlying systems of relations — onto schemes of growing com- 
plexity. The two paths are apparently disconnected, but they often draw from the same 
knowledge (the progress of geometry, optics, etc.) and sometimes they cross in ‘places’ 
such as, for example, memory theatres and, as it will be proposed at the end of this paper, 
potentially elsewhere today. 


Theoretical and Representational Issues 


The critical and theoretical literature on both of the aforementioned paths is growing 
[3]; but while the first takes advantage of its many connections with media studies [Grau 
2003; Zucconi 2018, pp. 149-181], the second is still struggling to find its roots mostly on 
marginal grounds [Facchetti 2019]: despite the fact that we are more and more exposed 
daily to infographics and data visualizations when it comes to Al [D'Abbraccio, Facchetti 
2021], we inevitably face the irrepresentable. Even our interaction with Internet and com- 
puters often appears mysterious, their responses being ruled by apparent randomness or 
by non-human wills. In our era, characterised by the availability of large amounts of data, 
machines, and algorithms capable of correlating them, one can easily yield to the temp- 
tation of proclaiming “the end of theory” [Anderson 2008] and give up building models 
that cannot compete with the ostensible objectivity of data and machinic intelligence. 
For instance, Google Translate has been using since 2016 a neural network developed by 
Google Brain to replace the previous st 


tatistical inference model. No more cross-refer- 
ences of texts, but a real intelligence: “not a set of two-dimensional connections between 
words, but a map of the entire territory [...]. The map is thus multidimensional, extending 
in more directions than the human mind can hold. As one Google engineer commented, 
when pursued by a journalist for an image of such a system, “| do not generally like trying 
to visualize thousand—dimensional vectors in three-dimensional space. This is the unseea- 
ble space in which machine learning makes its meaning” [Bridle 2018, p. 148]. 

Yet we are very aware of how much we need visual representations of complex phenom- 
ena per se and how much we need them to rule those phenomena. During the first stag- 
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Consider the pandemic graphs and maps [5]: the more data speak clearly 
r representations, the more they are insidiou 
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tions that are more difficult to understand and decipher 


s, to the point that they can be 


cases, paradoxically, the representa- 


are also the most engaging for 


those who want to understand them in depth, because they force the viewer to activate 
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tical awareness: they may not be adequate for quickly comparing trends — 


such as economic growth or the number of infections — but they work because they 
force us to pay attention to what is underneath. Common experience also shows that 
much of the work that careful use of these artifacts has to do, to find a thread to follow 
in such opacity, is in the reading of the texts that accompany visualizations, commonly 
found in scientific papers as well as in newspapers and magazines. Texts commenting 
visualizations are fundamental to understand and evaluate the choices made by scientists 
and/or infographic designers. 


Drawing at the Intersection of Digital Humanities and Human Digitalities 


The role of humanistic knowledge, therefo 
representation. Recently, the writer Helen 
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Dewitt and the statistician Andrew Gelman 
piece together [Dewitt, Gelman 2020] where they addressed the prob- 


we could call ‘DataViz criticism’ [6]. The article refers especially to the prolif- 


ualizations at the time of the Covid-19 pandemic, to their potential and 
plications and to the lack of attention given by designers to the possible 
uses of such graphic artifacts. The two authors begin by comparing an old diagram drawn 


Nightingale on the causes of mortality during the Crimean War (1853-56) 


rary analogous and much simpler diagram, highlighting how the first is 


more difficult to read but also more ‘engaging’ for the reader, more suitable for drawing 


her to a critical-narrative dimension. 
Drawing, between technical and humanistic knowledge, can be therefore 
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take part in the construction of new epistemological models: confronting the unfathom- 
able complexity of Als demands for new tools to build shared models and govern reality. 
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The alternative is to give up models and representations, to abandon science at the end 
of theory, to mere correlation among data [7]. Scholars are involved because they are 
familiar not only with technologies and workflows, but also with the optical, perspectival, 
geometrical and, more generally, representational knowledge that lies underneath, which 
is to say: with the very roots of the systems, infrastructures and interfaces that are shaping 
today's human interactions with the world. 


Between Perception and Scientific Analysis: Metaphors and -scapes 


As human beings, we probably need conceptual tools such as metaphors to deal with 
complexity [Bridle 2018, pp. 2-10]: after all, it's exactly what we do every time we use 
a desktop computer, open a folder, and edit a file. Today’s systems of power are often 
described by and modelled upon scopic regimes, as we have said about the potential of 
the digital twin of the world [Kelly 2019] and as we know very well from Michel Foucault 
[Foucault 1975], Martin Jay Jay 1988] and the last episode of Adam Curtis’ documentary 
series Can't Get You Out of My Head (BBC 2021). In fact, we are used to find metaphors 
coming from optics almost everywhere: transparency is a myth of modernist architecture 
as well as of interaction designers [8], but we claim it also from finance, economy, politics, 
at every scale. Catoptrics has been used by post-modernists to critique Modernism’s faith 
in transparency and to subvert it. And dioptrics may provide useful references to better 
understand and design today’s world with a critical attitude [Bergamo 2019], together 
with other optical notions such as, for example, that of parallax [Anderson 2018]. 

t is up to Drawing also the ambitious and relevant task of connecting and interweaving 
the paths of the two big categories with which this paper began: god eye’s maps and 
human eye's perspectival views. Some of the possible meeting points can be found in 
‘scapes’. Not only in the much-debated notion of landscape, which after von Hum- 
boldt merges the subjective perception of the observer with scientific collection and 
observation of information [Farinelli 2003, pp. 40-53]. But also, for example, in the many 
possible representations of soundscapes [Bergamo 2018] and in sound maps, which 
should start addressing the problem of placing perspectival and time-dependent au- 
dio field recordings onto interactive visual maps. And in datascapes, researched in the 
Nineties by the artistic collective Knowbotic Research and today by Ryoji Ikeda: abstract 
visualizations of digital data become a sort of artificial immersive landscape, engaging 
the viewer and, at the same time, potentially providing her with new visual metaphors, 
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new tools to represent what exists and design what will come. 


Conclusions 


Technology has evolved, the world has changed, but we are still the same species that 
tried to visualize and organize such a complex thing like memory in machines that worked 
ike modern theatres, after all. 

Drawing always played a major role in modeling knowledge. This paper tried to explain 
why the technical achievements it is deploying while facing the current challenges should 
be connected with the historical and critical background provided by the genealogies 
of perspective and maps: this much needed approach could lead to innovation in rep- 


resentation in today's world, and it is up to the Drawing scholar. 


Notes 


[I] George Wong in Frumkin Rebekah (2019). What the Scientists Who Photographed the Black Hole Like to Read. In The 
Paris Review online edition, www.theparisreview.org, (10 March 2021). 


[2] We could think about Paolo Veronese's frescoes at Palladio's Villa Barbaro as pictorial devices to “augment” the architectural 
experience of the owner and inhabitant and to “immerse” him in a virtual extension of the outer landscape, but the genealogy 
of this approach could go back in time much more, passing through Pompeii and up to cave paintings. 


[3] A book about the genealogies of both is e.g. Arcagni 2018. 
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[4] See https://eventhorizontelescope.org/press-release-april- 10-20 | 9-astronomers-capture-first-image-black-hole (10 
March 2021). 


[5] Some media designed their own, as in the case of The New York Times; others, the most, drew from renowned research 
institutions; expert users could also rely on more sophisticated online tools that allow to compare and filter data according to 
he answers they needed to retrieve. See https://ourworldindata.org/coronavirus (10 March 2021). 


[6] This phrase is not found on their article, but | think that it describes in synthesis what is at stake here. 
[7] Which may lead to aberrations, such as nonsense and funny correlations between the number of people who drowned 


falling into a pool and the films Nicolas Cage appeared in. See the website Spurious Correlations: http://www.tylervigen.com/ 
spurious-correlations (10 March 2021). 


[8] A good interface is one the user doesn't realize it's there, according to most and especially after Donald Norman. See 
orman Donald (1998). The Invisible Computer Why Good Products Can Fail, the Personal Computer Is So Complex, and 
nformation Appliances Are the Solution. Cambridge (Mass.):The MIT Press. 
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REPRESENTATION CHALLENGES 
Augmented Reality and Artificial Intelligence in 
Cultural Heritage and Innovative Design Domain 


Artificial Intelligency, Big Data and 
Cultural Heritage 


Giorgio Buratti 
Sara Conte 
Michela Rossi 


Abstract 


In recent decades, the cultural heritage sector has benefited from solutions offered by ICT for the 
conservation, management, enhancement and communication of cultural heritage; today this specific 
sector benefits from the infinite potential of application of Al. The proposed research identifies the 
three main lines of research that operate in the cultural heritage exploiting the synergy between 
machine learning, big data and Al, starting from the analysis of the state of the art and a subsequent 
first taxonomic approximation of artificial intelligence systems. The analysis of some case studies 
developed in the field of recovery and restoration of cultural heritage, monitoring and prevention of 
damage, data acquisition and analysis of the same, confirm the real potential of Al: trigger knowledge 
from knowledge. 


Keywords 
Al, cultural heritage, interaction, automatic drawing, taxonomy. 
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Introduction 


Since the 2000s, the improvement in microchips’ computing performance has made creating, 
collecting and transmitting data so fast and cheap, to bring the amount of available informa- 
tion to exceed a crucial threshold [1], determining the advent of the so-called Big Data era. 
Originally, the term “Big Data” referred to the technical ability to collect, store and process 
increasing amounts of data at decreasing costs. There is nothing innovative about this defini- 
tion, which could be compared to the advantages of printing over handwriting or any other 
technical improvement in cultural technologies history. The unprecedented aspect is the ease 
of accessing more data than it seems possible to handle for the first time in human history 
data is abundant and cheap and continues to exponentially increase. The information collected 
in digital form in a delocalized telecommunications network is accessible in real time to various 
types of electronic devices that allow data re-processing. In this context, any search creates 
in a few seconds a redundant mass of information that complicates the identification of the 
most significant content. The quantity and type of information has reached a complexity that 
transcends a direct management possibility, requiring an adaptation of the taxonomic ordering 
structures used up to now to understand phenomenal reality and to manage knowledge. 
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IA TASSONOMIA 
AI and Big Data: Order Complexity 


Unlike humans, computers can use the growing amount of ‘disordered’ information by 
defining a new way of producing knowledge, no longer based on reference theories, 
but on a posteriori identification of recurring patterns within the rough data flow. 
This abdication of the causal logic linked to reasoning in favor of computational anal- 
ysis does not imply a ‘technological’ determinism or that data driven research must 
guide all cognitive processes, because the data has no intrinsic value but in a defined 
context. If knowledge's inferential processes start from the informations acquired, 
it's also true that a simple data accumulation does not generate knowledge without 
a method capable of identifying the correlations among variables. In this context 
becomes crucial research on Artificial Intelligence, a definition that includes a differ- 
entiated set of techniques and methods with the aim of automating human cognitive 
abilities. And since computers process data at speeds now close to the speed of light 
[2], it is possible to analyze increasing amounts of information, generating predictive 
models based on objective assessments. Since its inception, AI research focuses on 
the creation of models that simulate the dynamics of human intelligence, based on 
the comparison and reworking of external stimuli, which can be classified according 
to the main active research directions. The table (fig. 1) shows on ordinate axis the 


abilities of human intelligence: 
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e Perception: skill to transform sensory inputs into useful information); 

e Reasoning: skill to connect / solve problems; 

e Communication: ability to elaborate a language and transmit concepts; 

e Knowledge: skill to understand and represent reality; 

e Planning:skill to define and achieve objectives. 

The main research lines currently recognized are summarized on the abscissa axis: 

e Symbolic Approach: which considers intelligence as that process of transforming perceptual 
stimuli into symbols to which a meaning can be attributed, according to more or less formal rules; 
e Statistical Approach: based on logical-mathematical tools to extract information content 
useful for transforming information into knowledge starting from Bayesian inference, formaliz- 
ing the hypothesis that a greater use of a given data increases the probability of its exactness; 
e Sub-Symbolic Approach: based on neuronal functioning whereby information is transmit- 
ted regardless of meanings: knowledge emerges regardless of previous mental models, but 
on the basis of structures based on experience. 

From these line of research, six subgroups descend which identify the main Al applications: 
e Logic-based: tools used for problem solving through the orderly representation of knowledge; 
e Knowledge-based: tools with extended ontologies and relational databases of notions, 
information and rules; 

e Probabilistic Methods: tools that allow agents to act even in incomplete information scenarios; 
e Machine Learning: tools that generate data from other data; 

e Embodied Intelligence: engineering toolbox, which assumes that a body (or at least a par 
tial set of functions such as movement, perception, interaction, and visualization) is required 
for higher intelligence 
e Research and Optimization: predictive calculation tools that allow you to optimize possi- 
ble solutions according to the specific context. 
These methods are already used, often unknowingly, on a daily basis: when we ask for 
directions to our portable device or ask a virtual assistant to turn on the light, we are 
interacting with natural language interpretation Al and neural networks trained to carry 
out research and home automation operations. 

For some years now, the use of genetic or evolutionary algorithms in form finding studies has 
consolidated the Al tools also in the drawing discipline. These processes consider the different 
morphological hypotheses as biological individuals and recombine them, similarly to sexual bio- 
ogical reproduction, introducing elements of disorder that simulate random genetic mutations. 

Thousands of formal solutions (new individuals) are thus generated, among which those 
that guarantee the highest level of optimization are chosen, just as selective processes 
promote the survival of the individuals most suitable for an ecosystem. In this way it is 
possible to explore solutions hitherto unused due to the lack of description and control 
tools adequate to the problem complexity. 


The New Frontiers of Al Applied to Cultural Heritage 


The first one collects all the research characterized by the development of innovative meth- 
odologies and systems to monitor and prevent damage to the cultural heritage, helping to 
increase the offer for the users. The most common example is the biosensor that can pre- 
vent damage from biological pests. This is any analog or digital device that works with living 
organisms (indicators) able to respond to certain environmental stimuli, producing an elec- 
tro—chemical, optical signal to reach a threshold value. Today automatic biosensors are studied, 
that work in real time and continuously to signal in indoor environments the achievement of 
favorable conditions to germinative factors and therefore a constant monitoring of the state 
of health of the good [Danese et al. 2019]. Significant are the researches based on the use of 
neural networks in conservative perspective; in particular the one carry on by the research- 
ers of National Gallery, Duke University and University College of London, that used neural 
networks to analyze complex high-resolution digital X-ray images of the cathedral of Gand 
“Adoration of the Mystical Lamb” [Van der Snickt et al. 2020]. Convolutional neural networks 
(CNN) combined with multi-resolution imaging techniques, initially used to determine the 
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The fields of application of Al are constantly evolving and searching for new contexts in 
which the application of predictive algorithms allows the extrapolation of finalized solutions 
from the automatic reprocessing of a large amount of data. The intelligence of the machine 
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ilities; in fact the 


described context opens to a series of reflections within the transformations taking place in 
drawing and in the contribution of the machine to design. 

For example, the Drawing-Bot is an Al developed by Microsoft in collaboration with three 
American universities that uses the technology of the antagonistic generative network to 
create images from its description. A neural generative network creates new images from 
some keywords, while an antagonistic network compares them with a database of existing 
images, eliminating the results that differ from it.At present this Al produces good results for 
short and explanatory expressions; it is able to easily extract the forms and then integrates 
them autonomously with the absent details from the given description, while the final result 
is likely to be a confused image for complex descriptions, but it is only a matter of time and 
speed of calculation. In the example [Zhang et al. 2017] the quality of the representation 
is close to the photographic one, but the reproduced bird does not exist in the reality, is a 
representation born from the “imagination” of a computer This case opens up to a series 
of reflections on the machine's learning potential and on how the relationship between the 
machine and the designer may evolve in the future. Today the machine is able to create a 
drawing based on the same geometric rules used to trace it by hand, but will it be possible 
to teach the machine the capacity of autonomous reasoning and to simulate the human 
creative capacity, through data processing? 


Notes 


[I] At the time of writing [2021] the amount of data currently engaged in the world each year is about 40 ZB [zettabytes]. 
One ZB corresponds to 102121 bytes, equivalent to approximately 180 million times the information collected in the records 
ept in the Library of Congress in Washington, recognized as the largest existing library with over 158 million documents held. 


[2] At the time of writing, the most powerful computer in the world is Summit, developed by IBM for the ORNL, United States 
Department of Energy, capable of handling 100 petaflops, equivalent to 100 million billion operations per second. Summit is 
used in studies ranging from particle physics to weather/climate forecasts, from the creation of new drugs to the simulation 
of the effects of nuclear explosions. 


[3] Although the paper was conceived jointly, Michela Rossi is the author of Introduction, Gorgio Buratti is the author of Al and 
Big Data: Order complexity and the related image, Sara Conte is the author of The new frontiers of Al applied to cultural heritage. 
The Conclusions are drafted jointly. 
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Abstract 


Nowadays museums realities are increasingly interested in introducing advanced technological tools, 
in order to preserve and enhance the Cultural Heritage, as well as to make it more accessible and 
attractive. In fact, digital tools, such as Virtual Reality (VR) and Augmented Reality (AR), allow to pro- 
vide effective responses to the needs of different types of users, to enhance the expressive qualities 
of the cultural asset, as well as to enrich the visitor's learning possibilities. However, the integration 
of new technologies in the management of Cultural Heritage involves different type of ethical issues. 
The paper analyses some of the ethical issues related to ontological, economic, and social aspects, 
trying to outline some principles able to limit these critical implications. 


Keywords 
museums, ethical issues, accessibility, virtual reality, augmented reality. 
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The introduction of these kinds of technologies could have different purposes. First, they 


could supply a significant contribution in improving the comm 


unication. The tou 


r experiences 


in historical, archaeological and museum sites could be enriched thanks to the introduction of 


multimedia contents, able to implement the visitor's 


earning possibilities through immersive 


experiences. The introduction of these technologies can be a way to increase the amount and 
the quality of information and to differentiate them by age and specific interests. The creation 
of innovative and amazing scenography (i.e. 3d reconstructions) and the introduction of dy- 
namic information transmission systems [Vaudetti et al. 2013, p. 95] could allow to increase the 
attractiveness the cultural goods for a wide audience, even non-specialists. 
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Fig. |. Recostructive 
proposal of the 
amphitheatre of Lecce. 

A frame of narrative 
approach — DiCet project 
[Gabellone 2015]. 


In addition to the communicative one, there are other purposes, equally important. In some 
cases, the virtual reconstruction of the Cultural Heritage could allow the virtual access of 
spaces that currently cannot be opened to the public for security or conservation needs. 
Within this frame, the intervention would also be a trace of a contemporary operation on 
the heritage, in compliance with the restoration’s guidelines of compatibility and minimal 
intervention. Furthermore, the need to preserve the historical-artistic memory of Cultural 
Heritage, constantly subject to degradation phenomena caused by natural and anthropic 
events (wars, terrorist attacks, etc.), is ever more urgent [Orlandi et al. 2014]. The digita- 
lization of Cultural Heritage could be an opportunity for passing on historical memory 
to future generations. Moreover, the life cycle of Cultural Heritage could be extended by 
limiting the accessibility to the sites and promoting exclusively (or partially) their digital use. 


Ethical Issues 


As seen, the use of VR and AR, as tools for the protection, development, knowledge, and 
fruition of Cultural Heritage, on one hand, achieves numerous goals, collecting wide inter 
est and success of the public; on the other hand, it raises some ethical questions, mainly 
related to the ontological, economic and social sphere. 

The first question concerns authenticity. In the context of the reconstruction through VR 
and AR of a setting (fig. |), a monument, or an archaeological object, we must ask our- 
selves about the authenticity of reproduction. In the case of virtual restitution of an object 
or a context that presents multiple gaps, the result irremediably presents a character of 
subjectivity, although the choices can be supported undoubtedly by an in—depth historical 
and scientific investigation. The outcome is the product of a univocal interpretation of the 
past through one of the many possibilities of analysis. In a try to bring the visitor closer 
to an in-depth knowledge of an object, in reverse, there is the risk of turning it away 
from the original, devaluing its value [Styliani et al. 2009, p. 525]. Therefore, the boundary 
between reality and fiction arises as to the main theme in VR and AR use. The informative 
and playful dimension must not risk compromising the authenticity of the work. 
Secondly, we must ask about the economic feature of this technology. Design, configura- 
tion, and maintenance of VR and AR systems usually have very high costs; their use often 
requires large spaces dedicated, and qualified personnel who take care of their opera- 
tion and use by the public. Therefore, it is an instrument in the economic possibilities of 
wealthy museum realities, able to sustain the technological investment; realities already 
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started and recognized, which — through their revenues or the support of patrons — im- 
plement the development of a heritage that may already be widely known, promoting 
its fruition in an innovative, diversified, modern, and endearing way. In the case of small, 
ess fortunate museums, with reduced spaces and limited resources, the adoption of this 
technology, for example, would make accessible more objects normally not exposed due 
to lack of space, as well as it would offer more attractive visiting ways, welcoming broad 
public interest and obtaining profit and sustenance. An economic gap emerges in the 
potential of these instruments: they are promoted by already virtuous realities and hardly 
available to others that would need them to promote themselves. 

Finally, a delicate question about social context arises. On one hand, the VR and AR use is 
extremely inclusive, providing opportunities to people with special needs (visual, acoustic, 
vocal, motor disabilities, and learning difficulties); however, on the other hand, the devices 
currently in use allow a type of visit characterized by a strong individualism. The isolation 
in the experience, often combined with the absence of a guide, can entail a loss of inter 
est by the user: the deprivation of a stimulating relational exchange — between visitors 
or with an expert guide — risks to reduce the educational value of the contents offered 
as well as the indispensable emotional dimension, despite the strong visual impact of the 
context. Moreover, the growing diffusion and application of these technologies in daily and 
domestic life, from cinema to videogames, implying the possibility of seeing without imag- 
ining, can induce an estrangement of the user from the essence and the purpose of the 
experience itself: the significance of the artwork, the scientific value of the reconstruction, 
the informative and didactic goal. Therefore, we witness the feared «Guggenheim effect» 
[Carrozzino & Bergamasco 2010, p. 457]: the fascination of the container overcomes and 
reduce the content, transforming the experience into mere entertainment rather than an 
opportunity to transmit information. 


Conclusions: Principles for Alternative Scenarios 


The use of these technologies, therefore, entails great advantages but also raises questions 
that lead us to reflect on principles able to limit their critical implications. 

Firstly, the sources on which to build the virtual reading of the Cultural Property must be 
undeniable, as a guarantee and protection of its values of authenticity and uniqueness. 
Secondly, from the point of view of communication, technology must act as a means of 
knowledge and culture, not as a mere attraction in order to implement tourist and econom- 
ic flows. The main purpose should be the communication of historical, artistic, and cultural 
values of a population, as well as the authenticity and uniqueness of a work. Therefore, it is 
necessary a deep awareness based on education and culture: both on the part of the de- 
velopers towards the contents to be communicated, and on the part of the users towards 
the added value that augmented reality offers. VR and AR are also instruments whose use 
must be calibrated and measured on the monument itself; cultural heritage institutions must 
assess their actual necessity and usefulness. 

Concerning the economic gap that underlies the possibilities of each museum, we have 
to reconsider the criteria for access to funding and redefine the distribution of resources 
for the design and development of these technologies. In this way, less fortunate museums 
could benefit from funding for the conservation and enhancement of their heritage, provid- 
ed that the proposals to be financed are examined by a multidisciplinary scientific commit- 
tee that controls the methodological validity and the ethical implications. 
Finally, technological development must increasingly be directed towards collective in- 
clusiveness, preferring opportunities for comparison and evolving from the individu- 
al dimension. The correct communication of scientific contents must be guaranteed 
through the interaction with a guide or through the presence of an interface that can 
offer support for the visit, hoping that the technological innovation may soon lead to 
social and increasingly inclusive use. 
n conclusion, VR and AR represent interesting tools to share knowledge, conservation, and 
development of the values of Cultural Heritage, provided that both their planning and fru- 
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ition are based on awareness and respect for uniqueness, materiality, and cultural, historical, 
and artistic values. There must be rules that control their use informing users that these 
technologies are not games: they share contents and information of high cultural value. The 
relationship between man and machine and the concepts of real and virtual must be kept 
at the center of innovation and application in this field. The many opportunities and risks 
impose that society negotiates the uses of these technologies, guided and supported by a 
coherent philosophical approach [Arcagni 2016]. In this way, these tools will become part of 
a historical moment and will represent a cultural value of our century to be handed down. 


References 


Arcagni Simone (2016). L'etica della realtà virtuale. http://wwwilsole24ore.com/art/tecnologie/20 | 6-05-09/l-etica-realta- 
virtuale-151037.shtmRuuid=AD4sn7 (February 2021). 


Carrozzino Marcello, Bergamasco Massimo (2010). Beyond virtual museums: Experiencing immersive virtual reality in real 
museum. In Journal of Cultural Heritage, | 1,2010, pp. 452-458. 


Felline Francesca (2015). Tecnologie innovative per la fruizione immersiva ed interattiva dei beni culturali. http://www. 
cniscintille.it/tecnologie-innovative-per-la-fruizione-immersiva-ed-interattiva-dei-beni-culturali/ (March 2018). 


Gabellone Francesco (2015). Integrated Technologies for Museum Communication and Interactive Apps in the PON DiCet 
Project. In: De Paolis Lucio Tommaso, Mongelli Antonio (eds.). Augmented and Virtual Reality. AVR 2015. Lecture Notes in 
Computer Science, 9254. Cham: Springer, pp. 3-16. 


http://www.arapacis.it (May 2021). 


http://www.eheritage.org/museum-installations/ (February 2021). 


http://ec.europa.eu/research/social-sciences/index.cfm (February 2021). 


http://www.progettoitacha.it/partner (February 2021). 


http://www.researchers.polito.it/success_stories/progetti_metti_in_rete/nuove_tecnologie_per_le_collezioni_museali 
(February 2021). 


Orlandi Marco,Vazzana Antonino, Zambruno Simone (2014). Tecnologia, Beni Culturali e Turismo: i Tour Virtuali Virtual Tours) 
come strumento per una corretta comunicazione dei Beni Culturali. http://patrimonioculturale.unibo.it/sge/?page_id=829 
(March 2018). 


Styliani Sylaiou, Fotis Liarokapis, Kostas Kostsakis, Petros Patias (2009). Virtual museums, a survey and some issues for 
consideration. In Journal of Cultural Heritage, 10, 2009, pp. 520-528. 


Vaudetti Marco, Minucciani Valeria, Canepa Simona (2013). Mostrare l'archeologia — Per un manuale / atlante degli interventi di 
valorizzazione. Torino: Allemandi&C. 


Authors 
Marco Ferrari, Dept. of Architecture and Design, Politecnico di Torino, marco_ferrari@polito.it 
Lodovica Valetti, Dept. of Energy “Galileo Ferraris”, Politecnico di Torino, lodovica.valetti@polito.it 


Copyright © 2021 by FrancoAngeli s.rl. Milano, Italy Isbn 9788835 125280 


39 


REPRESENTATION CHALLENGES 
Augmented Reality and Artificial Intelligence in 
Cultural Heritage and Innovative Design Domain 


The Magnificent Al & AR Combinations: 
Limits? Gorgeous Imperfections! 


Claudio Marchese 
Antonino Nastası 


Abstract 


Time and Space persist, in the digital age, as coordinates in which human presence is located. We 
will analyze some forms in which it is demonstrated “scenarios” concerning: a) spatial prefigurations 
of human ‘refuges’; a transition place between external and internal. Phenomenology experienced, 
in moving through the atrium of Alberto Campo Baeza's school in Loeches, crossed by the pupils. 
Movements punctuated, in opposite directions, by natural light variations and related shadows that 
continuously reconfigure it. AR b) measurements, regarding the astral motion of a satellite and shad- 
ow creations, revealed through light features. Variations defining the possibility in measuring time, due 
to three “bodies”, one that illuminates, another that is illuminated, and an intermediate one, receiving 
the shadow. c) relationships, where Exalt and Hologram, evoke in AR sequences the preservation of 
species, in which we are already protagonist, and in '2001:A Space Odyssey’, struggling with Al. 


Keywords 
prefigurations, measurements, relationships, space, time. 
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Fig. |. 3D Animation, 
elaborated by Antonino 
Nastasi. 


matic film sequence, is an emblematic case which recognizes the values mentioned above 
and testifies the designer's work experience who wisely displaces the light sources, captur- 
ing it within the architectural space continually reconfigured. The cyclical daylight passage will 
restore to the students, who occupy the space of the school hall, two significant moments 
of their day: entering and leaving the school. The rain of light coming from the large circles 
piercing the foyer ceiling, as the sunlight intensity varies, accompanies two moments of the 
student movements in the atrium space. In going in at school, the light captured at dawn, 
not very intense, escorts the students towards the interior creating a mandatory condition 
for concentrating on study activities. In coming out the luminous intensity of the midday sun 
announces the outside both with the disappearance of the ceiling due to the sun at the 
zenith conveying a great intensity of light and with the dislocation of the windows that focus 
the external ground, horizon and sky, respectively from the upper, intermediate and lower 
exit floor of the atrium. This declares the resumption of the relations with the materiality 
of the world, due to cognitive essence elaboration mediated by collective activity and pro- 
moted learning. 


Measurements 


An apparent star orbits the sphere, Earth. Moving along the trajectory of its orbit, relating 
the planet to the system that both belong to and from which receives light. A dynamics 
obtained by imposing laws of movement in an adequate program reproducing, in simulation, 
a portion of the Apparent Solar System complexity. 

This allows us to view from the outside and in a faster way, the effects induced by the es- 
tablished relationalities, in the subsequences of the reciprocal recurrence displacements. It is 
thus possible to ‘observe’ from a privileged point of view, putting our eye outside the system 
in which we are immersed, the dynamics of the apparent reality produced on our planet by 
the mutual dislocations of the spherical bodies mentioned above. It emerges, from the study, 
how the same exact configuration repetition of the two bodies, in relation to each other, 
occurs within cyclical cadences, determining variations that affect the temporal illumination 
quantities of the planet portions. The Earth, receives on its portions the shadow cast, illumi- 
nated by the very distant Sun, from which the light emanates. The ‘imperfection’ of the orbits, 
not circular, but elliptical, determined by the gravitational forces, generating the perpetuation 
of the movement, with the functioning of the gravitational forces, as a sort of springs that 
cyclically load and unload, allows, in a system equipped of substantial energy permanence, 
to cyclically replicate the reciprocal configurations passing through the ‘infinite’ others, fol- 
lowing one after another. Spheres, representing stars and planets, moving away and getting 
closer, reveal the formation of particular conditions, in relation to another ‘non—regularity’, 
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Fig. 2. 3D Animations, 
elaborated by Antonino 
Nastasi. 


due to the different inclination of the rotation axes of the planets. This demonstrates, in our 
virtual model of the apparent celestial motions, an unequal distribution of light as well as 
shadows cast on the various portions of the Earth’s surface. Something that we have been 
aware of for thousands of years and of which, thanks to computer science, we can now 
simulate with adequate authenticity. This allows to represent the apparent celestial sphere in 
which we are immersed, through the perceptual measurements of it. Unique measurements 
identify days in which the amount of light and darkness are equal (equinoxes) and two other 
days, in which respectively, lightness lasts longer and conversely less (solstices). The empirical 
model displays how these phenomena are the manifestations of the apparent world, as a 
result of the ‘rotations’ of the planets around their axis and of the ‘revolutions’, consisting in 
the elliptical orbit path movements. The Earth respect to the Sun and the Moon respect to 
the Earth. The video animation facilitates the understanding of shadow projections within 
the theory of conic sections. On one hand an imaginary shadow cone is cut by the plate of 
a sundial, while on the other the shadow of a sphere is projected on a plane demonstrating 
the Theorem of Dandelin—Quetelet. An aid of great importance, the scientific knowledge 
combined with empiricism, determined for example many actions carried out in agriculture, 
classifying the most propitious periods for sowing, other practices inherent to cultivating, as 
well as the most appropriate time for harvesting. 


Relationships 


A big wheel, the orbiting space station and the explorer spacecraft ‘dance’ in Stanley 
Kubrick's ‘2001 A Space Odyssey’, to the notes of the Straussian ‘Blue Danube’. Choosing 
this captivating sequence, fragment of the narrated story, it is undertaken a reflection on the 
relationship between man and artificial intelligence. 

The success of the mission is entrusted to the artificial intelligence of HAL 9000, the super- 
computer supporting the human crew of the Discovery, as well as the management with 
the scientist's hibernation on board, during the journey in approaching Jupiter The explora- 
tion was decided after the discovery in a Moon crater of a monolith, clearly extraterrestrial. 
Enigmatic, the monolith, which sends and receives signals from Jupiter The true mission 
essence, started only for exploration by the crew members, is acknowledged by the com- 
puter on-board that has ears and eyes throughout the ship, interrelating with humans. Such 
was implemented in order to make the on-board computer, among other things, also the 
human guarantor, a sort of their wet nurse. But, every time you rely on something, you also 
lose a little bit of your individual freedom, ending up at the mercy of it. Now, in coincidence 
with a damage repaired with an external human intervention, another malfunction relegated 
the astronaut outside, with his small hull, whose destiny ended up in getting lost in space, 
once the oxygen had finished for him, losing the commands. Only one astronaut remains in 
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Fig. 3. The frame is 

taken from the site: 
fantascienzaitalia.com. 
Page dedicated to the 
film 2001:A Space 
Odyssey by S. Kubrick, 

in occasion of the 50th 
anniversary re-release in 
theaters. 


the spaceship with active vital functions, who must program HAL 9000's rejection in order 
to revitalize the scientists, and reach the site where the exploration should be undertaken 
by the entire crew. The human man, drawn by this refusal, understands the danger constitut- 
ed by the ‘protective’ protocol used by the computer and decides to deactivate it. Therefore, 
engages a fight of unequal cunning with HAL 9000. If it wasn't for the human factor of the 
non-linear intelligence endowment, this allows him to have the upper hand in disabling the 


memory cards. Thus significantly regresses HAL 9000, verified by humming a childish song. 
The first information was taught to him by his inventor. It was exactly the contradiction be- 
tween the task of collaborating with humans and the incongruous prohibition, that provided 
the true crew mission ending in disrupting HAL 9000, and with what could have ensued, if 
not disabled by man. 


Conclusions 


(Prefigurations) It is clear, now, how the authors feel that the poetry of architecture can be 
unacknowledged, consequential of the augmented reality bombardment potentialities, care- 
fully avoiding from becoming sorcerer's apprentices. Certainly good designers will be able to 
grasp the possibilities offered by today’s most performing technologies, and are waiting so 
that these potentialities decant in order to configure the new changing life scenes. 
(Measurements) We should improve these studies and make them more effective to define 
better our well-being, in relation to animal species, as well as with vegetation and even with 
the mineral world, taking advantage of the time lap manifestations of the space portions. 
Therefore, continuously through Space and Time measurements, in which we are immersed. 
Relationships) The topic is clearly highlighted long ago. Respect to today the risk emer 
gence is even more evident, recommending a very supervised use of artificial intelligence. 
The enormous mass information in which artificial intelligences are continuously self-in- 
structed, reach incongruous actions, treating the data in statistical form that generate a 
secluded perception with the serious damage in dismantling human critical conscience. 
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Abstract 


Advances in information and communication technologies unveil unprecedented opportunities in the 
documentation and analysis of architectural and urban assets. Nevertheless, the (over—)abundance 
of data does not always seem sufficient for a thorough understanding. This contribution summarizes 
three connected research experiences that produced apps and prototypes exploiting geospatial da- 
tabases, Al for image recognition, and AR based on model tracking, applied to built heritage contexts. 
The three cases contribute to a reflection on the interpretation of the form of the built environment 
as a means to produce shared knowledge and operational outcomes. 
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Introduction 


Much of the narrative about technological advancement associates the collection of exten- 
sive and precise data with the quality of our understanding of architectural and urban assets. 
Augmented reality (AR), artificial intelligence (Al), digital survey, and digital modeling nourish 
this need for information. On the one hand, these rapidly evolving fields allow for unprec- 
edented and fascinating overlaps and synergies. On the other hand, curbing the hype that 
surrounds them is often needed. New technologies require attention to their fallibility, to the 
clarity of the purpose for which we use them, and to the generated models, which mediate 
our understanding of the documented objects. For example, the problems related to the 
classification of physical assets affect many tools — and in particular to those of Al, capable of 
manipulating abstract concepts. As data increase, who chooses the used categories, and how 
can these be adapted to each application that involves the tools? 

This contribution collects some ‘fieldwork’, that is, experiences in prototyping with tech- 
nologies to which, increasingly, we entrust the interpretation, classification, automatic repli- 
cation, and visualization of the built space. The projects described constitute the stages of a 
branched but coherent research path started at the ICEA department of the Universita di 
Padova and carried on at the Politecnico di Torino, through the activities of the FULL labo- 
ratory and with the collaboration between the author, Roberta Spallone and Marco Vitali at 
the Department of Architecture and Design. 

The first project concerns the development of a web platform, based on a geospatial da- 
tabase, and designed to guarantee access and interconnection to the multimedia results of 
architectural and heritage research. The project fostered a reflection on how even the vast 
digital information topology can benefit from the physical set of sources and their spatial 
characters. The second project is part of a study on the vaulted atriums of baroque Turin, 
which produced a catalog of over 70 sites and analyzed the geometric components of the 
vaults through digital models. The analysis and representation work was integrated with an 
AR application to visualize the geometric models superimposed against the images of the 
real vaults. Besides, we designed the system interaction with the web database built during 
the first cited project. The third project develops a mobile application that allows access 
to data on monuments by framing a target object with the camera and exploiting machine 
learning technologies for image recognition. This conjunction between Al and the built space 
aims at making cultural heritage more accessible thanks to low-cost, non-perishable, and 
easy-to-use infrastructures. 

The research path is an opportunity for reflecting on the processes that, from techniques for 
the recognition of spatial features — including Al and AR —, produce a schematic understand- 
ing and operational uses of the built form. 


First Project: a Flexible Digital Archive 


The first project, entitled Tu-Cult, lasted one year, was carried out at the Universita di Pado- 
va and was directed by Luigi Stendardo and Andrea Giordano [1]. The project produced a 
digital infrastructure on which subsequent experiences have relied, and involved several aca- 
demic and industrial partners and eight researchers in different architecture-related fields — 
including information modeling, geomatics, history, representation. The project featured the 
study of two churches in Padova, a small sample made significant by the differences between 
the two cases and conceived as an incremental contribution to a work on other monuments 
already started by part of the team [Cecchini et al. 2019]. 
The team aimed to enhance cultural tourism by means of information and communica- 
tion technology (ICT). This objective was pursued through three different kinds of outputs: 
(1) ‘data’, including documents, raw data, and all the other principal information sources; 
(2) ‘models’, that is, 3D models, information modeling, and other data processing products; 
(3) ‘visualization’, that is, tools and applications to deliver data and models to the final user 
through visual interfaces. Therefore, a significant part of the project dealt with the quantity 
and heterogeneity of the possible interpretations that characterize built heritage — which the 
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different outputs and professionals involved were able to grasp — and specifically addressed 
the role of representation as a means for sharing knowledge [Giordano 2017]. 

The team produced historical and documentary research, detail models and reconstruc- 
tions of historical phases, BIM models and digital survey models, visualizations in VR and AR. 
Intending to manage and connect these products, we set up the [cult] platform (fig. |, top 
left) [2]. It is based on a geospatial database and a set of web services, and makes the project 
information accessible and connected, even for external applications. The system does not 
underlie a complex hierarchy — it collects the entries into three simple categories: texts, im- 
ages, and models — but is based on ease of use and flexibility that have been verified during 
subsequent projects. The platform was built only with free and open-source software: the 
web interface was programmed with the Django framework, while the archive is a Post- 
greSQL database with the PostGIS spatial extension. Metadata are organized based on the 
Dublin Core standard for resource description [3]. Other technical aspects of the project 
are described in previous publications [Cecchini et al. 2019]. 


Second Project: Augmented Reality and the Built Environment 


The second project was started at the Department of Architecture and Design of the Po- 
litecnico di Torino. The research explores AR applications to the architectural environment 
and the possibility, already introduced in the context of cultural heritage, of making digital ar- 
chives accessible from physical objects. The research team — composed of Roberta Spallone, 
Marco Vitali and the author — had previously employed AR to connect physical archives and 
databases, within a project on the Theatrum Sabaudiae, catalog of the Savoy family residences 
at the end of the eighteenth century [Palma et al. 2018]. The result was an application an- 
choring small-scale 3D models to images. This experience evolved into a project to share the 
results of a study on over 70 Turin baroque atria of the seventeenth and eighteenth century, 
previously conducted by Spallone and Vitali [2017] [4]. Advances in AR allowed a shift in our 
experiments towards the architectural scale — even if, in early 2019 (the start of the project), 
commercial AR technologies mainly supported small-scale applications [5]. 

We developed an app to view models of the geometric interpretation of the atrium vaults 
(fig. |, bottom). 
The tests were conducted on a sample of four atria. The objects tracked for AR activation, 
were three-dimensional objects, of limited extension with respect to the entire environ- 
ment, such as the base of columns, or capitals. The targets were first registered as sparse 
point clouds and finally recorded in the app to enable runtime recognition. Since AR tools 
use many sensors (such as inertial platforms) to track the changing position of the mobile 
device, the small-sized targets allow visualizing the entire superimposed vault models with 
fair accuracy — even when the target is no longer framed. Finally, we verified the compatibility 
of the documentation on the atria with the [cult] platform, intending to use it as the database 
to be navigated from the AR view. In this case, we used the Xcode programming interface, 
a commercial software allowing free development. Other aspects of the project have been 
explored in previous publications [Palma, Spallone, Vitali 2019]. 
This ts still an ongoing project since the digital archive based on the atria has yet to be pro- 
duced. Furthermore, in the current state of technological advances, new solutions should be 
tested to use AR in the built space. 


Third Project: Deep Learning for the Recognition of Architecture 


The third project was conducted at the FULL Interdepartmental Center of the Politecnico 
di Torino [6]. The project aimed to optimize the resources invested in managing BH and the 
related information, using ICTs. Many solutions already in use at tourist sites (for example 
audio guides, or self-guided tours with QR codes) still require expensive and perishable in- 
frastructures and ad hoc projects. We have therefore created a solution for connecting the 
physical environment to the virtual space of information, documents, and digital models, ex- 
ploiting the capabilities of Al, the developments of which are starting to show the application 
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Fig. |. Top left: the [cult] 
digital archive, public 

web page. Bottom: 

model tracking AR; target 
acquisition as a sparse 
point cloud (left) and 
superimposed model in 
the final app, as tested at 
Palazzo Carignano, Torino 
(right). Top right: main 
views of the monument 
recognition app. 
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potential in the built environment [Pezzica et al. 2019]. In particular, we exploited a class of 
deep learning (DL) algorithms that allows image recognition [Goodfellow, Bengio, Courville 
2016]. Therefore, we have created an app that connects to the [cult] database and makes 
it accessible by pointing the camera of the mobile device at a Monument (fig. |, top right). 
The Al system evaluates the frames recorded by the camera and proposes a forecast of the 
observed object. Then, the app shows information related to the identified monument, such 
as descriptive texts, images, interactive 3D models, the position on the map, and other data. 
For a first experiment, the system was applied in the Central Archaeological Area of Rome, 
where it allows the recognition of 46 monuments that are very different in shape, size, and 
state of conservation. 

A significant step in building an Al system for image recognition is collecting the dataset for 
the training, that is, the process in which the parameters of the recognition algorithm are 
optimized for the required task. For this purpose, 50 to 100 photographs were collected for 
each monument to be recognized, choosing many different viewpoints on the object. The 
result is an engine that takes up about 13MB of disk space and can distinguish up to hundreds 
of objects (as an estimated magnitude) from every angle. The aspects that specifically con- 
cern the construction of the DL model are described in previous publications [Andrianaivo, 
D'Autilia, Palma 2019]. 
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Conclusions 


Technologies that interact with big-data risk replacing the work of interpreting the built en- 
vironment and the relationship that binds us to it, be it about culture, fruition, or design. We 
must prevent a general understanding from being neglected. Theory and models are needed 
to achieve also detailed data—driven knowledge, to value documentation and collection efforts, 
and to clarify the purpose of these efforts. The presented series of knowledge access models 
intends to support the identification of some contexts and scopes that can benefit from exper- 
iments with AR and Al technologies. We investigated how digital techniques for spatial charac- 
ter recognition support a theoretical understanding and operational uses of the built form. On 
the one hand, we must strive to keep the landscape of these possible uses broad. On the other 
hand, even the schematic understanding of reality that tools produce — the categorizations, 
the standards — should be kept flexible. Otherwise, automatic recognition and classification 
become the well-known black box, that is, algorithms we accept the results of without ques- 
tioning how these are produced or whether they can be adapted to changing needs. Finally, 
experiences suggest that, beyond the schemes we adopt, the tangible aspects of architecture 
remain a platform for effective knowledge sharing even in the virtuality-pervaded reality. 


Notes 


[1] The project was funded by the European program POR FSE 2014-2020 — Regione Veneto. A. Giordano was the principal 
investigator. L. Stendardo supervised the construction of the [cult] platform. 


[2] ICEA department, UniPD (2017). [cult] — culture containers. http://cult.reload.dicea.unipd.it (15 March 2021). 

[3] DCMI (2021). DCMI: Home. https://dublincore.org (15 March 2021). 

[4] In addition to the work of prof. Spallone and prof. Vitali, the research on the vaulted atria benefited from the collaboration 
of M.C. López González (Universitat Politecnica de Valencia, project “Nuevas tecnologías para el análisis y conservación del 


patrimonio arquitectónico” funded by the Ministry of Science, Innovation and the University of Spain) and Ph.D. students G. 
Bertola, F. Natta and F Ronco. 


a 


[5] Currently, some AR software such as Vuforia allow the tracing of very large environments based on CAD models or point 
clouds. See PTC (2020). Tools Overview | VuforiaLibrary https://library.vuforia.com/tools/overview.html (15 March 2021). 


[6] The team working on the project at FULL | The Future Urban Legacy Lab consisted of M. Robiglio, C. Casetti, F. Frassoldati, 
L.N. Andrianaivo e V. Palma. The project was carried out in partnership with R. d'Autilia. 
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Abstract 


The research analyses the evolution of automatic drawing technology, starting with some singular me- 
chanical experiences, such as the puppets by Pierre and Henry-Louis Jaquet-Droz and Henri Maillar- 
det up to developments in graphic technology for digital drawing, such as the ones by lvan Sutherland, 
Timothy Johnson and Nicholas Negroponte. 

Finally some potentials in the use of intelligent algorithms are described, showing how it is possible to 
use shape grammar for architecture, as the experiences by Hank Koning and Julie Eizenberg on Frank 
Lloyd Wright's Prairie houses demostrated. 
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Introduction 


‘ 


Drawing Automata’ is an ambiguous title. It can mean at the same time ‘to draw automata’ 
and ‘automata that draw’. But if in the first case the meaning is perfectly in line with the 
provisions of a certain discipline, the drawing one, in the second case we are in the pres- 
ence of a phrase that exceeds this work, requiring additional skills, different from those of 
a good designer who, holding a pencil, draws lines on a sheet of paper 

Here we are interested in deepening in particular the second meaning we have indicat- 
ed, namely that linked to the themes of Virtual Reality and Augmented Reality which are 
among the topics addressed in this symposium. ‘Drawing Automata’ has something to do 
with Virtual Reality. In both cases, we can recognize an oxymoron, that is a rhetorical for- 
mula that undoubtedly captures attention by ‘combining antithetical terms’. On the one 
hand, in fact, there is the term drawing, which each of us associates with one of the most 
traditional and creative human activities; on the other hand, the term automata brings to 
mind the world of the machine — more or less intelligent — which, autonomously from 
human will, exercises its role in total independence. There is a large variety of automata: 
from the robots operating in modern factories, to the human-like replicants in the film 
Blade runner by Ridley Scott, which in the future probably will be used also in our houses. 
Nicholas Negroponte reminds us that phrases such as Virtual Reality and Artificial Intelli- 
gence, to which we could add our Drawing Automata, can be considered oxymorons — as 
we have indicated above — but at the same time also pleonasms, because they amplify 
the contents they describe, enhancing their semantic value [Negroponte 1995, p. 116]. 
We could integrate the list with the term Augmented Reality which, even more explicitly, 
exemplifies the idea of pleonastic enhancement that we described earlier 

We must not overlook the fact that Negroponte himself had underlined the epochal 
change that took place thanks to technology: from the world of atoms, in fact, starting from 
the mid-twentieth century to the world of bits, that is to say to the physical recording of 
information to the digital archives [Negroponte 1995, p. | 1] . This current condition sees 
an exponential multiplication of the bits stored in digital memories, so much so that the 
neologism Data-ism was coined as a meaningful way to describe the period in which we 
ive. The term was used for the first time in a text by David Brooks on The New York Times 
[Brooks 2013], but it has been taken up more extensively into a book by Yuval Noah Harari 
[Harari 2017, p. 449], in a chapter titled The religion of data, in which the author analyzes 
the epochal change in society, thanks to the invasion of data in our daily life. 


Brief History of Automata 


We find the term “automata” in one of the first treatises on machines written by Heron of 
Alexandria, who lived in the first century AD. Published for the first time in 1589 [Herone 
1589], the treatise contained no reference to drawing machines, but was full of mechan- 
ical tools, some of them have become everyday used. These machines are reminiscent of 
those developed by Leonardo da Vinci, such as the beautiful gears in the manuscript of the 
Codex Atlanticus [Leonardo da Vinci 2006], drawn with great dexterity and skill already a 
century before the publication of the translation of Heron's treatise. It should not be for 
gotten that in Leonardo's library [Vecce 2019] there were some transcriptions of previous 
codices — such as Taccola's De ingeneis [Vecce 2019, p. 91] — from which Leonardo himself 
certainly took inspiration to draw up his extraordinary graphic drawings as well as some 
of those that are counted among his inventions. Think for example to the parachute, which 
the artist proposes in the manuscript mentioned (f. 1058v) in the form of a pyramid and 
which appears in a manuscript of an anonymous Sienese engineer in the form of a cone. 
The first mechanical hand is found in the surgical treatise Les Oevres d’Ambroise Paré [Paré 
1579], published in 1579. It is called “iron hand” (fig. la) by the author who describes it in a 
precise manner, with the aim of generating a prosthesis for the human limb. Represented in 
all its details, it recalls that of the “draftsman” (fig. |b) created by two watchmakers, Pierre 
and Henry—Louis Jaquet—Droz, a little more than a century later with the intention of gen- 
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Fig, |. from left to wright: 


a) the “iron man” by 
Ambroise Paré [Paré 
1579]; b) Pierre and 
Henry-Louis Jaquet- 
Droz, the draughtsman; 
c) Henri Maillardet, the 
draughtsman—writer; d) 
some drawings by the 
Maillardet's automaton. 


erating curiosity on the part of bystanders [Perregaux 1906, Marchis 1994]. It is a puppet, 
that draws mechanically and infinitely the same drawing, because its moving mechanisms 
are regulated by time gears, reminiscent of those of a clock. It is no coincidence that at the 
same time the physician Julien Offroy de La Mettrie concluded his book entitled L'Hom- 
me Machine with the sentence: “the body is but a clock” [La Mettrie 1747]. Even today it 
is possible to see the mechanical doll intent on drawing if we go to the Musée d’Art et 
d'Histoire in Neuchatel, where the restoration of the mannequin made it possible to com- 
pletely restore the object's operation. In reality, in those recursive drawings there is nothing 
creative: the mechanism is similar to that used by the first pen plotters of the 1970s which, 
through a pointed graphic tool, drew lines on the sheets, avoiding the method of inkjet 
used today. But it is interesting that the drawing is made by an automated instrument and 
not by a human hand. 
Similar to the preceding one is the mechanical puppet realized by Henri Maillardet, who 
worked in the Jacquet-Droz laboratory [Ceserani 1969, pp. | 16-1 18]. Maillardet's autom- 
aton (fig. lc) was even more sophisticated because it could be programmed to generate 
different drawings (fig. | d). So not just one, but several ones, as if it were controlled by 
an algorithm in which it was possible to modify certain parameters in order to return 
different results. About these automata, Thomas A. Heppenheimer wrote that “in these 
inventions we can discern many of the features of today’s programmable industrial ro- 
bots” [Heppenheimer 1985, p. 42], although we have to wait a few years before we see 
interesting results in the field of intelligent algorithms. 


CAD and Al 


To find experiences that deal with computer machines that produce graphic draw- 
ings we have to get to the 60s of the twentieth century, with the experiences of Ivan 
Sutherland [Sutherland 1963] and Timothy Johnson [Johnson 1963], authors of the first 
interactive tools for CAD drawing, in the first case two-dimensional, in the second 
already three-dimensional, a few months after the first. It is difficult to think that this 
synthesis could take place without the theoretical support of an authoritative figure 
such as Steven A. Coons, to whom we owe the first algorithm for the generation of 
complex surfaces, which take their name from him. Even today, many modeling soft- 
ware describe such forms as Coons Surfaces, an explicit reference to the author who 
described the algorithm in a well-known essay [Coons 1965]. Only a few years after 
that experience, Sutherland developed the first virtual reality system — which will be 
called The Sword of Damocles, borrowing the name from the well-known legend told by 
Cicero. The principle proposed in Sutherland's system — called by him also the Ultimate 
Display [Sutherland 1965] — is contained in every current virtual reality system, even if 
it presents a system similar to what is now called mixed reality, with the union of real 
vision with virtual vision. Stereoscopic glasses therefore offer the observer a virtuality 
that is perfectly superimposed on the actual vision. 
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Fig. 2. from left to right: a) 
. Negroponte, system 
that recognizes shapes 
and constructs 3D model 
[Negroponte 1975]; b) 
H. Koning, J. Eizenberg, 
Shape grammar applied 
to Prairie Houses by 
F.L Wright: Stiny house, 
itchell house, March 


ouse [Koning, Eizenberg 
1981]. 
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To find the use of artificial intelligence, it is necessary to wait few years, in which shape 
recognition tools are developed. This is the case of some experiences described by Ne- 
groponte in two books, The Architecture Machine [Negroponte 1970] and Soft Architecture 
Machines [Negroponte 1975], where graphic recognition algorithms are proposed, which 
can include graphic signs and return consequent three-dimensional shapes (fig. 2a). But the 
most interesting experiments will be the ones carried out in the field of shape grammars, 
and in particular those in which the role of architecture is involved. We thing, for example, 
at the experiences conducted by Hank Koning and Julie Eizenberg to develop methods of 
analysis on the works of Frank Lloyd Wright, thanks to which to re-propose architectures 
in imitation of those of the American architect. So three new Wrightian houses were de- 
veloped (fig. 2b) to which the two researchers gave the name of three important figures 
in the filed of computer science, whose studies were decisive in the development of the 
investigation of architecture using new technologies: George Stiny, William Mitchell and 
Lionel March, helping to define the process of digital semantic analysis of architecture that 
is now implemented in many studies. We can remember the ones conducted at Massa- 
chussets Institute of Technology by José Pinto Duarte (Pinto Duarte 2001) on the work of 
Alvaro Siza Vieira, in which the shape grammar of the Malagueira's houses was studied to 
understand the way in which the architect used the composition rules to design his work, 
proposing new possible variations of them. 


Conclusions 


The archives of stereometric forms, investigated in a timely manner with syntactic and se- 
mantic analysis tools, can constitute a morphological database from which useful informa- 
tion for the project can be drawn, both by human designers and by intelligent algorithms 
that can select and propose solutions suitable for different purposes. Precisely this new 
methodology of intervention gave rise to an acute debate to understand the thin border 
between human creativity and computational intelligence. 
The breakthrough on the table of the designer of the automated drawing is ultimately 
modifying the architect's work — i.e. we think to the Building Information Modeling process, 
in which clash detection algorithms can help architect to find problems during the creation 
phase — so much so as to arouse enthusiasm and anxiety at the same time.A recent essay 
by Mario Carpo — one of the most authoritative figures in the field of criticism of new 
technologies for the architect — clearly expressed this status, underlining how develop- 
ments in algorithmic design can find unthinkable ways out. This reflection is well expressed 
in one of sentences by him — which closes the cited essay — which summarized the concept 
with a highly communicative hyperbole: “Architects cannot do without technology, but 
technology can do without them” [Carpo 2018]. 
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Abstract 


The links between representation and artificial intelligence (Al) invade many fields of architectural re- 
search, recording continuous and significant advances: they require, on the one hand, a constant update 
of the state of the art and, on the other hand, careful consideration of the role of Representation in 
interdisciplinary research in this field. 

The present contribution intends to investigate these intertwining in some of the most frequented 
research fields in recent years: the valorization and communication of Cultural Heritage and cultural 
tourism, the experiences in the museum field, the research on the role of the prototype within the 
processes of artificial intelligence applied to architecture. 
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AR technology [Sandor 2015].The group coordinated by Geronikolakis (2020) proposes an 
interesting application in the field of cultural heritage conservation to improve the experi- 
ence of visitors, involved in the construction and restoration of archaeological sites. 

In the field of Al, the representation disciplines could mainly be involved in the outputs of Ob- 
ject Recognition operations. The applications of this technology involve the user also outside 
the museum, as in the project ‘Recognition’, winner in 2016 of the IK award [Styx, 2020] or the 
application ‘Art Selfie’. Several museums base the exploration of their collections on Object 
Recognition: on color similarities, [Cooper Hewitt Smithsonian Design Museum, n.d; Dallas Art 
Museum, n.d.], formal, line direction, or space and light [Barnes Foundation, n.d]. 


Alamuseum 


Fig. la. Exibition Mont 
Saint-Michel. Digital 
Perspectives on the 
odel (photo by 
icrosoft, source: https:// 
iciseattle.com/en/mont- 
saint-michel-in-mohai/); 
Fig. |b. The system of 
CAR-CNR group 
experienced in art 
and tech exhibitions 
Caggianese et al., 2020]. 


Fig, 2. Project 
FLOWMORPH [Hahm et 
al. 2019]. 


In recent years, technologies have made considerable prog 
tailored to the needs and interests of different types of users, 
centered approach. However, t 
accessibility 
The Al context is ce 
of AI/AR/VR technologies re 
of an increasingly wide 
The exhibit 


ress in the creation of content 
with a democratic and human- 
here are still few examples where Al has been used to foster 
and inclusion in the field of museums and cultural heritage [Pisoni et al, 2021]. 
rtainly characterized by a high level of multidisciplinarity The mixed-use 
i presents an important field of investigation to meet the needs 
audience, from an inclusive perspective. 
ion “Mont Saint-Michel — Digital Perspectives on the Model” (fig. | a), held in late 


2019 at MOHAI (Museum of History and Industry) in Seattle, represents an example of the 
application of mixed techniques [Ici Seattle, 2019].Visitors were immersed in the history of 
Mont-Saint-Michel by framing a real scale historical model of the site with a mixed reality 
device (Microsoft HoloLens 
Another example in this frame is the work conducted by ICAR-CNR [Caggianese et al., 


2020] related to 


2). 


the display of Leonardo daVinci's machines (fig. |b). This presents the com- 


bined use of cutting-edge technologies, such as holograms, with artificial intelligence (Al) 


to offer the visi 


+ 


manipulation, and conversation). 


Al in Digital Fabrication Experiences on Architecture (GB) 


The link between real model, virtual reality and augmented reality is artic 
lines of research. The first one is based on the construction of informati 


tor an augmented space involving new forms of interaction (visualization, 


ulated on two main 
ve real models, the 


second one on the interaction ‘human—material’ through the practices of ‘augmented crafts- 
manship' and ‘design by making’. Regarding the first, many studies are now focused on the cre- 
ation of prototypes aimed at the transmission of knowledge and information of architectural 


her 
wh 


With 
peri 
involved the setting up of a s 


model, conceived as a ‘wunde 


ence 


ble 


+ 


n recent yea 


tage. In the last years, this has happened both through the realization 
ich to apply immersive technologies, and with more complex dynam 


+ 


of static models on 
ic systems In which 
physical and digital models have come to overlap to the point of almost confusing. 
regard to the application of immersive technologies, reference can be made to the ex- 


carried out for the Basilica of Loreto [Rossi et al. 2017, pp. 239-255].A project that 
pace consisting of a multimedia table with a 3D printed physical 


rs, however, rapid advances in techno 


+ 


rkammer equipped wi 


th a series of technological devices refera- 
to different applications characterized by different levels of interactivity and immersiveness. 
ogy have begun to challenge even those 


aspects that go beyond the mere static and physical representation of architecture. An 


example is the interactive model of INFORM [Foll 
shape display proposed by 


different 


+ 


ways of physical int 


+ 


mer et al. 2013, pp. 417-426] a dynamic 
the Massachusetts Institute of Technology (MIT), that suggests 
eraction in real time between users, physical model and digital 


data. Regarding the ‘human—material’ interaction, the main fields of investigation concern 


the 


vative s 


ITK 


of comp 
at investi 


app 


+ 


ication of biological 
tructural morphologi 
E Research Pavilions are a series of full-scale prototypes realized through the integration 
g, advanced analytical techniques and digital fabrication aimed 


utational engineerin 


gating the architect 


Regarding the same themes, 


es through digital fabrication methods. In particular, the IC 


ural potential of fiber composites [Doerstelmann et al. 201 


62 


principles on architectural construction and the study of inno- 


D/ 


5]. 


in London there are also the Protohouse project and the Flow- 


Morph project [Hahm et al. 2019, pp. 553-562] (fig. 2), that proposes an unconventional 
method of fabrication using Mixed Reality to materialize highly complex geometries that 
could not be realized manually or by robotic fabrication alone. 

The practices of digitally augmented craftsmanship open to a series of reflections on the 
theme of interactive digital simulations and their link with traditional craft practices. Mario 
Carpo, within his book The Second Digital Turn [Carpo, 2017], emphasizes the value of 
such simulations as they allow today's artisans to learn intuitively, through mistakes, trial and 
attempts, by making and unmaking as many test samples as possible. 
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Physical and Digital Pop—Ups. 
An AR Application in the Treatises on 
Stereotomy 


Alessio Bortot 


Abstract 


This paper focuses on the relationship between stereotomic traces and building construction tech- 
niques. Starting from the first essential drawings representing stone cutting methods in full scale — 
which are still visible in some Gothic cathedrals in France and Spain — it will be analyzed the evolution 
of stone cutting drawings to the most recent examples in | 9th-century treatises. In particular, it will 
be examined the progress in terms of representation according to different centuries concentrating 
on the development of theory and practice. It will be therefore given a short overview of the most 
important Renaissance essayists dealing with changes in representation methods according to the 
codification of assumptions related to descriptive geometry. In the last part it will be proposed the 
involvement of augmented reality techniques in order to visualize and study Louis Monduit's treatise 
on stereotomy through the use of digital pop—ups. 
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stereotomy, Louis Monduit, 3D modeling, descriptive geometry, augmented reality. 
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Evolution of The Trait in the History of Stereotomy 


The research carried out in this paper has led me to study several stereotomic drawings 
in order to understand the reasons why these ‘blueprints’ are connected to the concrete 
practice of construction. The first stereotomic traces are engravings on the floor (less often 
on the wall) in some European and Middle Eastern Gothic cathedrals. In this regard Cal- 
vo-Lopez states “tracings were prepared exactly below the element under construction in 
order to control the placement of voussoirs. Generally speaking, in the Gothic period they 
were executed on scaffoldings placed under vaults, at springer level, while in the Early Mod- 
ern period, they were laid directly on the floor” [Calvo-Löpez 2020, p. 127].The large-scale 
drawings — which seem to transform the buildings where they are located in a sort of ‘stone 
treatise’ — were used to solve specific problems and allow the formal compliance of the 
piece under construction. In other words, these first stereotomic developments were not 
intended as a representation of an abstract problem and didn't want to spread construc- 
tion skills as well. In fact we should remember that in the Middle Ages stonecutters used to 
belong to guilds that usually concealed every building skill. As a matter of fact, stereotomy 
was a type of practical knowledge which used to be conveyed orally by the foreman to 
his stonecutters only in the construction site. In addition, it should be taken into consid- 
eration that writing and drawing materials were very expensive at the time. For instance, 
parchments were much more used in case of public presentations to clients than in stan- 
dard communication [Erlande-Brandenburg 1993, p. 79]. In this period the most important 
graphic document is the well-known notebook Livre de portraiture by Villard de Honnen- 
court (...— XIII century). Even if this latter can't be really considered a treatise on stereotomy, 
anyway gives us a meaningful example of representation in the field of construction site 
machines, building techniques and use of proportions in the Gothic period. According to 
Sakarovitch, it is quite interesting to notice that in the carnet by de Honnencour one can 
find some elements which are represented in double projection despite a lack of awareness 
in terms of a projective correspondence in the two views [Sakarovitch 1998, p. 41]. 

We know that stereotomy literature thrived in the Renaissance period basically due to Le 
premier tome de l'architecture (1567) by Philibert de l'Orme (1514-1570) [1]. In this treatise 
the illustrations show how the medieval secrets — traditionally kept by the guilds — were final- 
y unveiled. The stone cutting techniques spread at the same time as the role of the ‘architect’ 
flourished — intended in the modern meaning. From being a ‘mechanical art’ architecture 
was gradually turning into a ‘liberal art’. What's more, these historical dynamics pointed out 
the separation between the architect and the foreman (the maftre-macon), between the 
designer and the builder The authentic expressive medium of the building work is the trait: 
the technique allowing to trace the layout of the stone structures in order each ashlar to be 
properly cut. For what concerns this topic, Robin Evans (1944-1993) asserts that the trait 
is something that is in between two different roles: artisans on one side and architects on 
the other Although it isn’t part of any of them, it exists as an independent reference such 
as geometry [Evans 1995, p. 205]. In this period treatises are didactic tools, but also ways to 
show the skills of the authors, often very critical of the stereotomic solutions developed by 
their own predecessors or peers. In the treatise by de LOrme virtuous artifacts arise along 
with complex illustrations. One of the best examples is the renowned Trompe dAnet — a 
case mentioned a few centuries later by ViolletHe-Duc (1814-1879) as an “artifice that has 
nothing to do with the rigorous art of the builder made to amuse curious spirits with unnec- 
essary problems” [Viollet-le-Duc 1854-1868, book IX, p. 314]. The drawings of this structure 
(one in pseudo axonometric projection to show the object in athree-dimensional space and 
some others in pseudo orthogonal projection) aren't able to solve the problem except for a 
reader who is also an expert in construction site practices. Something similar occurs to other 
planches where geometric operations (similar to ‘surface unwrapping’) are applied to each 
ashlar row in order to obtain the corresponding épures. In Renaissance treatises it is likely to 
find a certain need for three-dimensional representations. Thanks to the codification of the 
perspective laws, in this period these models — which in the Middle Ages were kept secret in 
the constructors mind only — acquire education purposes as well. 
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Several treatises on stereotomy were published in the Enlightenment. This way, cutting 
solution were codified as well as graphic strategies developed to explain the techniques 
of division into ashlars. In the huge production of scientific books on this topic it is essen- 
tial to mention at least the ones by Amedee-Francois Frézier (1682-1773) [2] and Jean 
Baptiste de la Rue (1697-1743) [3]. Both are illustrated with high-quality figures using 
orthogonal projections, axonometric and perspective views of arches, vaults, trompes and 
staircases. For the first time in the history of treatises on stereotomy some parts dealing 
with the explanation of general geometrical problems (e.g. conic sections, intersection 
among shapes, etc.) are included too. This addition is quite important because it reveals 
a different approach to the relationship between form and structure which can define 
the superiority of geometry in the field of this construction technique. Other descriptive 
strategies used by de La Rue suggest the need for a ‘stereoscopic’ approach which gives 
the opportunity to go beyond the two-dimensional limits characterizing paper surface 
such as the use of pop—ups. In fact, his Planche XXXIII describes — both graphically and 
through pieces of paper which are glued to the page — the mistake made by de La Rue's 
colleagues in the solution related to the corner segment of a spherical vault. In this sort 
of stereotomic ‘origami’ the reader can concretely realize the error by de la Rue's prede- 
cessors which is explained developing the faces of the block of stone. More in general, 
we can affirm that the use of this technique comes from the necessity to control very 
complex formal configurations such as spherical vaults or trompes. 

In scientific literature the tradition connected to the use of pop—ups or movable books 
dates back to the medieval period [4]. Another example in this field concerns the use of 
removable paper elements (flaps) or the aforementioned pop-ups in treatises of gno- 
monics and perspective. Finally, movable books on optics, gnomonics and stereotomy 
became significant tools when simulating the projective and geometric processes applied 
to these disciplines, in order to support the representation of elements in space. What's 
more, three-dimensional paper models used to be designed in order to visualize the 
cutting flat surfaces of stone blocks in space in order to find the shape of the single faces 
and define the skin containing the volume. 


Louis Monduit’s Treatise: a Case Study 


| chose the treatise by Louis Monduit as a case study to carry out an experiment on 
augmented reality. His book was published in a historical period in which stone cutting 
construction technique was about to be substituted by new methods and materials 
which were soon the turning point in terms of development in the field of architecture 
in the 20th century. Stereotomic techniques were therefore strengthened and opti- 
mized due to experimentations and publications on the same subject over the centu- 
ries. The treatise by Monduit doesn't show any innovation related to cutting solutions, 
since the case studies included belong to a tradition which dates back to the Renais- 
sance period. Beyond, some features are particularly useful in this research such as the 
clear projective coherence of the drawings and the general structure that serve specific 
didactic purposes. This volume opens with a glossary of technical terms in the field of 
stereotomy and projective geometry. Monduit supplies some definitions referring to 
the measurement systems and concepts related to planar geometry. Then he draws his 
attention to the examination of the main proportional ratio, the study of solids, polyhe- 
dra (with their development) and a short (but precise) explanation of concepts related 
to descriptive and projective geometry. A special consideration is dedicated to the 
method of orthogonal projections which is explained first in three-dimensional space 
and then on plane. This first section is followed by the study of intersection among sol- 
ids (cylinders, cones and spheres) — represented in cavalier drawing. This specific part is 
clearly influenced by the well-established method which was codified by Gaspar Monge 
(1746-1818) — professor at the Ecole normal supérieure of Paris since 1794. At the end 
Monduit explains the tracing of the épures and the technical design of the project. Both 
are useful to develop every ashlar properly. 
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Fig. |. Exploded view 

of the 3d model 
obtained from the digital 
reconstruction of Planche 
46 of Monduit's treatise. 


decided to use digital tools for AR in order to reinterpret the need for three-dimensionality 
expressed by the historical treatises on stereotomy. The first phase of the work has involved 
the construction and research on the digital model of the case studies proposed by Louis 
Monduit.The 3D models were furthermore imported to a Unity software equipped with Vo 
furia Engine plug-in. As soon as every table in the treatise was set as a target recognizable b 
the mobile device to display the corresponding model, | defined ambient lights and materia 
to ensure a successful visualization. The following step involved the organization of content 
in the form of an app for tablets and smartphones designed to study the treatise through it 
digitalization and the definition of models with different semantic values. The different info 
mation can be found on the screen overlapped to the real book framed by the device.A first 
menu entitled ‘geometry and shape’ allows us to see the digital model divided into blocks, the 
essential geometric entities representing the cutting planes of the ashlars and the opportunity 
to observe the model through an exploded view drawing (fig. 1). 

For what concerns this device it will be possible to read the Monduit's text through the use 
of a parallel window in the interface which explains every geometrical phase in the defi- 
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Fig. 2. False color Fem 
analysis of the 3D model 
obtained Planche 46 of 
Monduit's treatise. 
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nition of various traits. Placing the mobile device in front of the book it will be possible to 
refer to a database of georeferencing associated with every single case study. This database 
will gather the existing architectures which were carried out and theoretically described 
by Monduit in order to obtain an extensible catalogue of stereotomic structures with a 
description of their own historical and artistic features. Anyway, starting from the important 
relationship between form and structure in the analysis of stereotomic buildings, | decided 
to add the opportunity to see a structural model in this app showing the strain of the ele- 
ments through the use of a false color Fem analysis (fig. 2). 


Conclusion 


This experiment has shown that the augmented reality can be considered an important 
tool in the enhancement, analysis and dissemination of cultural heritage through the cre- 
ation of digital pop—ups. Stereotomy can be considered an excellent field to experiment 
these kinds of technologies and highlight the closer relationship between geometry and 
structure while interpreting ancient and complex treatises. In addition, we can state that 
the use of these technologies must be combined with an in-depth critical development of 
contents as well. Otherwise the risk is to appear as simple tools of entertainment, instead 
of sources of knowledge. 


Notes 
[I] De l'Orme Philibert (1567). Le premier tome de l'architecture. Paris: Federic Morel. 


[2] Frézier Amedee-Francois (1737-1739). La theorie et la pratique de la coupe des pierres et des bois pour la construction des 
voûtes et autres parties des bátiments civils 8: militaires, ou Traité de stéréotomie, a [usage de l'architecture. Paris: Guerin. 


[3] De la Rue Jean Baptiste (1728). Traité de la coupe des pierres, où par une méthode facile et abrégée, l'on peut aisément se 
perfectionner en cette science. Paris: Charles-Antoines Jombert. 


[4] To analyzed this topic, please, see for e.g.: Candito Cristina (2018). Drawings and Models in English Perspective Treatises 


of the XVII and XVIII Centuries. In Cocchiarella Luigi (ed.). ICGG 2018-Proceedings of the | 8th International Conference on 
Geometry and Graphics. Milan: Springer, pp. 1882-1894. 
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The Value of a Dynamic Memory: 
from Heritage Conservation in Turin 
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Abstract 


The proposed contribution is the start of a research project on the conservation and care of the 
memory of urban contexts that are changing and that, while changing, it is desirable to preserve trac- 
es of past evolutions. These are particular areas of study, characterised by significant values that need 
to be known and passed on. Since the beginning of this year the research group has been dealing 
with significant experimental cases. On this occasion, some quick elaborations and prefigurations 
of documentary integration with the existing heritage will be presented for the case of the Borgo 
Crocetta nursery school building in Turin. 


Keywords 
cultural and environmental heritage, expeditious urban survey memory conservation, graphic codes. 


Oz Quartiere Crocetta 


(F. Barrera, R. Nelva, G. Novello) 


SCUOLA MATERNA BORGO CROCETTA 
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The Study of the Cultural Environmental Heritage of Turin (1984) 

The analysis of urban form and the architectural-environmental identity of the built city are 
themes addressed in various research experiences conducted in the Turin area by many 
well-known scholars. The experience of the census of the cultural and environmental heri- 
tage of the City of Turin has given substance to general urban planning instruments consol- 
idated for Turin in subsequent years and it is now being updated. 
n this contribution, we will look at an important aspect of this census, at how digital re- 
sources and archives keep alive its cognitive function, at some possibilities of integration 
that collaborate with the methodological application of the research begun in the 1980s, 
and finally, we will share some considerations that should support the continuation of our 
future work. 
At the beginning of the 1980s, on the occasion of the revision of its general regulatory plan, 
the City of Turin promoted research and studies concerning the complexity of its articulated 
urban system and the scientific approaches necessary to govern its transformation pro- 
cesses. These researches have been carried out by organisations of scientific importance 
and, in particular the Politecnico di Torino, under an agreement with the City, has identified, 
recognised, and classified the architectural and environmental cultural heritage of the entire 
municipal territory. 
The foundation of the identification and classification work was the regional town planning 
aw (LR. 56/77) and the preparatory studies for the town plan entrusted to Gregotti and 
Cagnardi (1995).A very broad research team, coordinated by Professor Vera Comolli, which 
mainly involved two Departments, one from the Architecture area, the House and City 
Department, and one from the Engineering area, the Department of Building and Territorial 
Systems; for the engineering area, the area from which we come, there were Pina Novello, 
Riccardo Nelva and Paolo Scarzella. 
As Vera Comoli wrote in introducing the study: “The research uncovers a very articulated 
and diffuse consistency of the environmental cultural heritage, much more extensive and 
analytical than the traditional binding recognition, much more differentiated according to 
cultural specificities, values, and implicit values” (AA. VV. 1984, p. 28). 

The methodological approach also involved the identification of a graph- 
ic language and representation codes based on municipal technical cartography. 
The work on Cultural and Environmental Heritage represents a very in-depth field of in- 
vestigation, especially in the reading of its relationship with the city. 

One aspect that has emerged is the importance of investigating ways of preserving mem- 
ory before intervention in areas of special interest transforms their functional and formal 
aspects while preserving the identity and recognised elements. In the current state of the 
art, it is of substantial interest to understand how to implement and update the information 
collected in the study, published in 1984, to make the permanence of those considerations, 
updated, accessible in a dynamic way within the information systems dynamically accessible 
within the existing information systems. 
Within the study merged in the publication, a different degree of compatibility with urban 
transformations was identified to define and order three categories of values linked to 
protection constraints graded according to the hierarchy of the specific urban quality: as- 
sets of historic-artistic value, assets of environmental and/or documentary value, signs of 
cultural and/or documentary significance or interest. Assuming the criterion of active pro- 
tection and environmental safeguarding and restoration for all categories of assets, for the 
for first two the type of eligible interventions is strictly limited. 
On the other hand, intervention is possible in areas of cultural and/or documentary interest 
segnalazioni). In fact, after exhaustive documentation and critical analysis, in-depth trans- 
formations can be prefigured, but “sectorial, casual, in any case disqualifying interventions 
cannot be accepted”. We are focusing our interest on this type of property as part of an 
ongoing experiment. 
We think that the mediation of traditional representation tools is experiencing a renewed 
development and breath through today's ways of support and treatment offered by the 
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Fig. |. The case study 

is based on the 

analysis of the available 
documentation. The figure 
shows a graphic summary 
of the contents on the 
Museo Torino portal and 
the Geoportal of the City 
of Turin. 


methodologies and technologies produced by information science; these allow many data 
to be organised using in-depth analysis and digital representations with different and artic- 
ulated levels of detail. 

The sphere of the ‘segnalazioni’ assets, for the type of interventions that it allows, makes it 
possible to exploit methodologies and technologies that make the paths of preservation 
of memory more agile and keep their presence dynamic; the mediation of the traditional 
tools of representation lives a renewed development and breath through updated methods 
for the support and treatment of data; methods that, through the technologies produced 
by the science of information, allow to organize many elements of knowledge with insights 
and digital representations with a different and articulated level of detail, integrated into the 
computer tools that institutions are increasingly sharing (fig. |). 


Methodological Aspects and Initial Scope of Application 


The development of integrated digital archives, in which easy access to all types of cul- 
tural heritage is made possible, with interactive and hypertextual navigation between the 
assets themselves, is a significant development. For their implementation, it is necessary 
to set up a digital inventory complete with databases of real and virtual assets, acquiring 
digital images with descriptive cards, creating digital images and virtual descriptions, and, 
where possible, inserting descriptions and three—dimensional representations of the asset, 
obtained with the most appropriate survey techniques. 

Through the selection of a first representative and significant case of the category of 
assets described above, it is possible to compare the most effective and efficient ways 
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of consultation, which are not limited to orderly cataloging but incorporate processes of 
an expeditious survey and automatic and ‘intelligent’ restitution (photo modeling). This is 
the step that has to be taken to support the management of a multi-relational database, 
whose information potential is increased by geometric-spatial and textual-numerical di- 
mensions, suitable to be implemented to define, record, disseminating, and processing 
data and information characterised by individual and contextual specificities. 

The ‘segnalazioni’ of urban areas of environmental and/or documentary interest in- 
dicates buildings and aggregates of the city which constitute significant evidence of 
a determined historical period and a particular cultural season or possess intrinsic val- 
ues for a physical and functional improvement aimed at raising the urban quality. 
In figure 2 some elaborations were carried out on the Borgo Crocetta school building in 
Turin, which represents an example of ‘segnalazioni’. 

The case study is based on the analysis of the available documentation and in- 
vestigates more interactive tools that bind knowledge within graphic containers. 
The knowledge supports for Turin have for years been stratified in photographic images 
and maps that are rapidly accessible and consultable, semantic digital archives, Plan in- 
struments and general urban planning instruments. 

The existing wealth of information can be supplemented by expeditious and digital sur 
veys, from the photo modeling of the context and of the building at the urban scale, to the 
instrumental detection of architectural details that can be properly interpreted by artifi- 
cial intelligence tools and then be integrated into digital information models. By definition, 
augmented reality offers users the possibility to integrate virtual objects in three dimen- 
sions into their living environment in real-time. But there are different ways to do this, in- 
cluding marker—based, or markerless solutions based on global positioning systems (GPS). 
Solutions based on ‘markers’, or on location or scene recognition, respond to the sen- 
sors of mobile devices and allow multi-dimensional information to be placed in a giv- 
en location. The stratifications of indications have followed different skills over time. 
Their sedimentation can be observed today in the information system of the City of 
Turin. In this case we observe a partial deficiency, missing an element of connection be- 
tween studies made, collected in part, and those currently present. In this sense, it will be 
necessary to pose the issue of criticism and self-criticism on the updates of the Cultural 
and Environmental heritage, starting from the registration of the size of the urban fabric 
while recognising the architectural and building value. Furthermore, it will be necessary to 
create consultation tools that are more ductile and easier to consult and update the data: 
census resulting from a survey linked to the design of areas linked to the Cultural and 
Environmental Heritage, producing materials according to standard protocols to support 
the administrations to verify the reuse of shared data and evaluate integrations. 


Concluding Remarks and Future Developments 


Integration into the multiplicity of levels referred to must be supported by appropriate dis- 
semination tools; the dimension of the mutations inherent in the same part of the territory 
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investigated in the future or documented through materials of different origin or temporal 
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derivation is a relevant aspect, just as the heritage of elaborations contained in the research 
should not be dispersed, or ignored, if we want to encourage an active and dynamic mem- 
ory for those significant parts of assets and contexts which, although modified or redefined 
by interventions made necessary over time, could be kept alive and present in virtual 
scenarios, in order not to forget their primitive value and integrating the different possible 
forms of reading and interpretation. 
The setting up of a recognisable and still valid method, whose application today is confront- 
ed with analysis and recording tools that are more agile and above all more accessible to 
diversified users with a high degree of accessibility, is today measured by the demateriali- 
sation of knowledge. It is necessary to recognise technologies that are easy to use so that 
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they can be applied widely and homogeneously, technologies that multiply, but above all go 
beyond, in the permanence of criteria and standards relating to the data and metadata to 
be stored: the ambition to ‘put everything together and in relation’ as an interpretation of a 
problem that derives from alternative technological knowledge that supports information 
intentions. In this sense, several questions will have to be addressed in the continuation of 
this research work which has just begun: How should information be managed, does it make 
the tools chase or does it chase them? How to ‘seed’ urban memory, material identifiers or 
global positioning systems? 

We are confronted with this: the transformation has taken place and that place today cannot 
tell its memory, its memory is mute or not everyone can understand it or can interpret it. The 
paradigm of what urban memory can lose and not transmit through the place itself, in the 
very place where it was produced, and not only through documentary heritage and archived 
knowledge, by their nature not accessible to a wide public, not consultable by walking those 
streets. One can only feel the distant sense of the human and urban reality that occupied that 
space in the past, but not visualise it, explore it at a distance of time. The seeds of memory 
need to be planted where memory is constantly being renewed, lost and reformed. The ma- 
terial that constitutes cities must therefore be used to narrate their transformation 
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Augmented Reality and the Enhancement 
of Cultural Heritage: the Case of Palazzo 
Mocenigo in Padua 
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Abstract 


Da palazzi nobiliari a sedi universitarie: nuovi modelli di studio e rappresentazione del patrimonio architet- 
tonico dell'Università di Padova is the large project which includes the study presented. The aim of the 
research is to investigate the virtual methods of access and enhancement of the Palazzo Mocenigo 
Belloni Battagia in Via Sant'Eufemia in Padua. The use of ICT (Information and Communications Tech- 
nology) is increasingly applied in the cultural heritage sector, especially in the context of museums, in 
order to increase their inclusiveness and raise the level of use. By exploiting these technologies and 
applying them to the case at issue, we will try to give visitors the opportunity to virtually see a building 
closed to the public. Inside Palazzo Mocenigo preserves precious frescoed halls, dating back to the 
Renaissance period in which the building was one of the major cultural centers of Padua. 
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virtual reality, augmented reality, cultural heritage, virtualization, enhancement. 


doi.org/ | 0.3280/0a-686. | 2 


Fig. |. AR simulation with 
Reality Composer. 

(digital model by Antonio 
Calandriello). 


The research project Da palazzi nobiliari a sedi universitarie: nuovi modelli di studio e rappre- 
sentazione del patrimonio architettonico dell'Università di Padova [|] was created to investigate 
some significant buildings received in the form of donations or acquisitions to the University 
of Padua. These are generally little-known noble palaces, compared to sites such as Palazzo 
Bo, Palazzo Liviano, but of great interest for the Paduan historical-architectural heritage, as 
well as for the history of the city and University. During the twentieth century, following the 
exponential increase in the student population, the university acquired real estate numbers 
of historical and artistic interest to use them as offices, faculties or student residences. Four 
buildings of particular importance have been identified in the project: Palazzo Contarini in 
Via San Massimo, Palazzo Selvatico Luzzato Dina in Via del Vescovado, Palazzo Sala in Via San 
Francesco and Palazzo Mocenigo Belloni Battagia in Via Sant'Eufemia. 
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As it appears today, Palazzo Mocenigo is the result of the unification of two distinct struc- 
tures, built at least in the fifteenth century but profoundly modified, and then connected 
between the sixteenth and seventeenth centuries. On the site of the current northern body 
stood the church of Sant'Eufemia which still today gives its name to the street, a very old 
ecclesiastical foundation. Some traces may remain in the hypogeum attached to the build- 
ing. It was a city quarter where other Venetian patricians settled, attracted by the proximity 
to the water that facilitated the connection with Venice, but also by the wide availability of 
green spaces that made it a pleasant location. 

The history of Mocenigo consists of several successive acquisitions, expansions and a re- 
structuring, promoted by Leonardo Mocenigo son of Antonio, which took place in the 
second half of the sixteenth century. The frescoes in some of the internal rooms of the 
souther wing — attributed to Gian Battista Zelotti (1526-1578) —, and the ceiling paintings 


80 


Fig, 2. Hypothesis of 

VR simulation with 
smartphone and Google 
Daydream View. The VR 
image shows one hall of 
Palazzo Mocenigo. 


commissioned to Stefano Dall’Arzere (1515-1575) for one of the rooms on the ground 
floor — that are now in the Civic Museums of Padua —, belong to this second phase of the 
work. From this moment Palazzo Mocenigo became a reference cultural center for the city. 
The building, now owned by the Belloni family, underwent further works in the first half of 
the seventeenth century, and other rooms were frescoed by Daniel Van Der Dyck (1610- 
1670). The building as it reaches us today is the result of further changes by successive 
families and in 1955 it was finally used as a residence for students in memory of Lina 
Meneghetti. The project was entrusted to Giulio Brunetta (1906-1978) who, in addition 
to the internal adaptation changes of Palazzo Mocenigo itself, adds a further body of the 
building. In 1960, the University of Padua also bought Palazzo Mocenigo building, which 
together with the Brunetta building, has been closed to the public since 2012 following 
the earthquake in Emilia Romagna. 


In the case in question, taking advantage of current digital technologies, also borrowed from 
other sectors that differ from that of cultural heritage, an attempt is made to offer a solution 
to the impossibility of visiting the property and at the same time, should it reopen the doors 
to public, offer an increased level of use.Within this process of enhancement it is necessary 
to underline how the use of ICT (Information and Communications Technology) not only 
allows a higher degree of use — enhancing the tangible and intangible — and improving social 
inclusion, but is also able to play an important role also in the conservation of the asset. If 
we consider the process behind an enhancement project, we will note that this involves an 
enormous acquisition of data — often of a multidisciplinary nature — which returns a photo- 
graph of the current state, both from the point of view of the conservation of the property 
and of the-state-of-the-art connected to it. It is in this phase that the digitization of the 
documents that are acquired and processed in order to be adequately disclosed takes place. 
The study for the enhancement of Palazzo Mocenigo was designed using different digital ap- 
plications and wants to be paradigmatic also for the other buildings that are part of the key 
project. Technologies used and intended to be used include virtual reality (VR), augmented 
reality (AR), video production and application development. 

VR can have different levels of interaction, a passive one in which the user 'moves' and an- 
other one extremely interactive that gives him the opportunity to interact with space and 
virtual objects. In the specific case, as will be seen later, we focus on a less advanced level. 
Augmented reality is a technology that allows you to view additional information in real 
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time through an overlap between real and virtual elements (dynamic audiovisual content, 


3D animations, films). Through special markers and a device, the real image is enriched with 


features and information that integrate and complete it. 
The video production, on the other hand, allows an informative experience through more 


sive experiences listed above. 


In order for augmented reality and vir 
create digital content to be subsequently transferred into the various software and appli- 
cations. The first step was to create a virtual clone of Palazzo Mocenigo, this happened 
two ways: with a digital photogrammetric survey and with 3D modeling. The digital photo- 
grammetric survey mainly concerned the internal environments — the halls richly decorated 


traditional channels, but at the same time it was the starting point for some of the immer- 
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tual reality applications to work, it is necessary to 
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with frescoes showing forest scenes with a Flemish taste and the rooms with suggestive 
painted architectural perspectives — were detected through high-resolution photographs, 
respecting the criteria of overlapping and spatial distribution dictated by ‘structures from 
motion’ software [2]. Particular attention was paid to continuously detecting the interior 
of the building, focusing on the connections between one environment and another This 
urn a unique model of the frescoed rooms and the con- 
necting rooms with the certainty that all the rooms were perfectly oriented and scaled with 
respect to the others. The information previously acquired and contained in the images of 
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approach made it possible to ret 
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the passage rooms, used simultaneously in different chunks of Agisoft Metashape, made it 
possible to automatically align the different models of the individual rooms (Workflow> Align 
Chunks> (Method) Camera Based). At the end of all the various phases for the production 
of the mesh model and its texture, this was exported to be then combined with the virtual 


clones of Palazzo Mocenigo. 


The digital models of the entire building were created through the use of historical doc- 
umentation as regards the one that shows the formal conformation of the building in the 
mid—| 6th century, while for the current state it was based on the digital photogrammetric 
survey and current documentation. The three-dimensional models, which form the com- 
mon basis of the various declinations of the enhancement project, were subsequently used 


in different ways. 


The structure of the multimedia project of Palazzo Mocenigo foresees as a first output is 
the creation a totem that will be located in front of the building itself and that represents the 
‘stargate’ for accessing multimedia contents that are recalled and displayed on their devices 
by scanning of specific QR codes. 
A first QR code offers the possibility to view a video that tells the story of the building, 
its historical evolution and the transformations it has undergone in relation to the urban 
context. Paying particular attention to showing what it is not possible to see today, revealing 
the contents of the internal rooms and passively guiding the viewer inside and outside the 
building, both in its current configuration and in that of the sixteenth century. 


A second tag allows you to download an application that contains a series of multimedia 
contents, including interactive ones. The application provides that the collection of scientific 
and educational material is arranged by simple accumulation, starting with the most generic 
information materials and then descending, based on the user's needs, into a series of in- 
creasingly specialized information. Through a ‘hamburger’ menu it is possible to access the 
different sections that include the historical documents of the land registry, the State Ar 
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chives of Padua and Venice, 


the historical information of the building and its transformations 


over the centuries, the building at the time of residence student, the 360 ° virtual tours of 
the various frescoed rooms, the historical information connected to them, and finally the 


+ 


building in augmented reality. Some of these experiences, as in the case of 360 ° virtual 
tours and the building in augmented reality, can be recalled individually by framing other 
special QR Codes placed on the entrance totem with your device. 

The idea is not to bind the user to necessarily download an application, but to equally use 
part of the multimedia content made available on online platforms such as Momento360 
[3] or to use applications already present in their device as in the case of RealityComposer 
or ARCore, respectively for iOS and Android platforms. The models of Palazzo Mocenigo 
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that should be shared on an opensource platform such as Sketchfab [4] and subsequently 
displayed directly in AR on their devices. The models created for augmented reality allow 
you to view the formal and distributive aspects of the building and the arrangement of the 
frescoed rooms in the two reference periods: at present and in the configuration in the 
sixteenth century (fig. |). 
The virtual tours have been designed to allow the user to mainly explore the interior of the 
building. Each 360 ° render allows you to explore the interior of the rooms in an immersive 
and interactive way (fig. 2). Thanks to tags placed on the different points of interest, the 
user can receive information about the paintings or simply move from one environment to 
another. The information may be of a different nature, from information about the author, 
the construction, the construction technique to more interpretative ones, such as for the 
study of architectural perspectives. This last case would represent the maximum expression 
of the potential of virtual reality applied to the field of Cultural Heritage. The user would 
find himself immersed in a completely artificial space that arises from the restitution of the 
architectures painted on the vaults and on the walls of the rooms. There would therefore 
be not only the possibility of visiting an inaccessible physical space, that of the rooms of 
Palazzo Mocenigo, but the possibility of finding oneself even in an different dimension, the 
one depicted in perspective on the walls and vaults of the building itself. 


Notes 


[I] Born to experiment new ways of studying, narrating and communicating the historical—architectural heritage received by 
the University of Padua in the form of donations or acquisitions, the research project started by an interdisciplinary team 
coordinated by Elena Svalduz as scientific director (Dbc), Andrea Caracausi (Disgea), Andrea Giordano (Dicea), Nicola Orio 
(Dbc), Stefano Zaggia (Dicea), involved some researchers (Antonio Calandriello, Simone Fatuzzo and Umberto Signori). The 
writer has dealt in particular with the enhancement project that is proposed here. 


[2] The software used for image processing was Agisoft Metashape Pro. 


[3] https://www.momento360.com 


[4] https://sketchfab.com. . 
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Abstract 


The illusion of ancient frescoes takes on a new form through the methods offered by digital technol- 
ogies. This contribution deals with the representation of a geometric solid in the decoration of the 
Room of Leda in Palazzo Balbi Senarega in Genoa: this is the small stellated dodecahedron described 
by Johannes Kepler in 1619.An in-depth documentary and iconographic research has allowed us to 
prove that this Is the first perspective representation of this solid which took place about thirty—five 
years after its discovery. The aim is to investigate and illustrate in a communicative, but not simplifying 
way, the connections between an example of Genoese decoration and the wider seventeenth-cen- 
tury milieu, which features artists, scientists, and generous patrons with their unsung brides. 
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stellated polyhedra, Kepler, illusory architecture, perspective, augmented reality. 
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Architecture and Polyhedra in the Room of Leda 


This contribution offers an updated representation of the history and meaning of some 
polyhedra painted in the Room of Leda, in Palazzo Balbi Senarega, in Genoa [|]. The Palace, 
which has been part of the University of Genoa since 1972, was built by Bartolomeo Bianco 
around 1616 for the brothers Giacomo and Pantaleo Balbi. In 1645 it passed to Francesco 
Maria Balbi who wanted to enlarge the building and, starting from 1655, hired the best 
painters and quadraturists for the decoration of the second noble floor 
The painterValerio Castello and the quadraturist Andrea Sighizzi created the frescoes in the 
Room of Leda which probably belonged to Barbara Ayrolo, wife of Francesco Maria Balbi 
who occupied the adjacent Room of Hercules. 
To identify characteristics and appropriate modes of representation, we carried out the 
following analysis [2]: 
|. Photogrammetric surveys and nodal photography. 
2. Geometric analysis of the represented perspective. 
3.Application of perspective restitution procedures. 
4. Material and documentary investigations on the polyhedra. 
5. Modeling of illusory architecture. 
6.Applications of Augmented Reality techniques. 
In the Room of Leda, the painted architecture replaced the actual pavilion vault with a gold- 
en oval vault, with a large central eye in which the myth of Leda and Zeus was represented. 
The intent was to fully integrate various forms of art according to the Baroque style. In the 
present contribution, we improved our investigations about the relationships between real 
and illusory space, starting from the perspective restitution procedures. In the absence of 
geometric-proportional references, they are based on the alignment of the projective lines 
from a point of view (V) generated by an observer in the centre of the room. 
The illusory architecture of the oval vault may have been inspired by illustrious coeval archi- 
tectures in Rome. The oval shape, perhaps prompted by the rectangular plan of the room, 
may refer to the vault of San Carlino alle Quattro Fontane by Francesco Borromini (1634- 
1644), whose shape is also echoed in the later Sant'Andrea al Quirinale (1658) by Gian Lo- 
renzo Bernini and in other famous planimetric shapes of sixteenth and seventeenth century 
architectures, as the helical staircases in Palazzo del Quirinale by Ottaviano Mascarino (late 
sixteenth century) and in Palazzo Barberini by Borromini (1625-1633). 
The profile of the illusory vault, in tts complexity, could be also be compared with Guarino 
Guarini’s vaults as the one in San Lorenzo (Turin, 1668-1687), with their oval openings and 
their balustrades, that in Turin is the base of the lantern and in Genoa prelude to the sky. 
The representation of architecture is attributed to the quadraturist Andrea Seghizzi as well 
as probably the geometric elements in the Room of Leda, such as the polyhedron, repeated 
six times. An in-depth documentary and iconographic research on the polyhedron allowed 
us to prove that it was the first perspective representation of one of the solids described by 
Johannes Kepler in 1619: the small stellated dodecahedron, which, together with the great 
stellated dodecahedron, implements the small array of the five regular solids, extending 
them to non-convex polyhedra. Recent researches on frescoes have also provided an op- 
portunity to obtain new insights into the relationship between art and science in a complex 
cultural environment of the mid-seventeenth century [3]. 


The Reconstruction of Illusory Space 


The implemented virtual model is a geometric interpretation of the architectural scene 
represented in the painting of the intrados of the vaulted ceiling of the Room of Leda. Once 
built the photo—modelling of the vault, the geometric reconstruction of the main architec- 
tural and decorative elements took place. 

As the first step, we realized a digital model of the room; in this way all the main real ar- 
chitectural elements (i.e. perimeter walls, doors, windows and the frame that run along the 
impost) configure the digital space in which the 3D photo—modelled mesh has been inset. 
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Once the real space is configured, we move on to the construction of the virtual space: 
namely the one depicted on the vaulted ceiling. The imaginary observer — who is standing in 
the center of the room — individuates the point of view (V): all the subsequent reconstruc- 
tion steps relate to this point, in order to achieve a straight geometric reconstruction. The 
point of view (V) is placed at 1.7 meters (hv) from the floor level, simulating the height of a 
human observer because we cannot establish a point of view through the internal perspective 
references. Therefore, a virtual camera is placed at this height and it is tilted zenithal upwards. 
n order to include the entire space of the vault in a single shot, we choose a wide-angle focal 
ength (about 17 mm). Here it is the first sensible approximation: as is known, wide-angle 
focal lengths deform the reality, especially in the edges of the frame. In fact, a focal length clos 
to a 50mm on a full frame camera should be used in order to obtain a photographic imag 
resembling the unaided eye view (the so-called “normal vision”). This focal length represents 
the optimum value, since it impresses on the frame an image similar to the one perceived by 
the human eye and it is therefore able to reproduce a correct proportion of the subjects 
according to all three dimensions simultaneously (height, width and depth). By accepting thi 
optical-photographic simplifications, the following 3D modeling procedures turn in a digital 
model the main architectural elements painted on the golden oval vault. Namely parapet, 
coupled lonic columns, medallions, frame, windowed vault, polyhedra and balustrade above 
the central oval oculus. By observing a symmetrical arrangement of the depicted scene, two 


orthogonal symmetry axes (x and y in fig, la) are defined overlying the hypothetical observer. 
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whole parapet model, the element is then mirrored according to the x and y axes previous- 
ly described. As a general rule a vertical arrangement of the elements was assumed. How- 
ever, we observed an incompatibility in the position of the bases of the columns compared 
to their capitals which are not vertically aligned. This misalignment was verified through the 
simulation of an ideal configuration; in the vertical alignment, in fact, the capital would be 
positioned in the annular extension of the frame of the vault. With the described structural 
issue, we also recognized a contradiction in the figurative superimposition of the frescoes. 
These were the reason why we adopted the choice of an oblique arrangement of the col- 
umns nevertheless recognizing its incompatibility with an ideal architecture. Furthermore, 
again to respect the result of the superimposition, an irregular shape of the oculi has been 
assumed which allows to visualize the cusp of the lunettes. 

The modeling of the virtual architecture represented on the vault of the Room of Leda 
therefore achieves a good approximation by returning a navigable digital environment where 
the user can observe both the architectural and the decorative elements. The six stellated 
dodecahedrons hanging with ribbons on the surface of the windowed vault are positioned 
in the six pendentives symmetrically. 


Augmented Reality Applications 


The digital model of the architectural space was implemented in its augmented reality 
version using Unity software (Unity Technologies) together with Vuforia plug-in, thus gener- 
ating the application to be installed on a mobile device to make visible the reference image 
and the three—dimensional model associated with it (fig. 2a). 

The available augmented reality technology definitely oversteps the two-dimensional rep- 
resentation's limits, providing a set of tools that allow overtaking the simple visualization 
by offering an immersive experience. AR tools expand the spatial investigation possibilities 
and they are at once resources for communicating the knowledge of the place which one 
comes in contact with. Neil Spiller, for example, used augmented reality applications in an 
unusually suggestive way within the Walled Garden for Lebbeus project [Spiller 2014]. In his 
personal and poetic eulogy to Lebbeus Woods, he overlapped virtuality on the image in 
order to inspire emotions: in this way he introduced within representation the environmen- 
tal factors (e.g. weather phenomena and phantasmagorical presences) affecting the spatial 
reading and that permit to investigate in more detail its representation. Augmented or 
mixed reality technologies can be used for dissemination purposes to take advantage of spatial 
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analysis features and streamlining the interaction process between the architectural environment 
and the viewer while making it intuitive and direct. The application of this systems allowed us 
to highlight differences between the unaided eye view of the Room of Leda frescoes and their 
virtual reconstruction. Moreover it was possible to focus on the understanding of the small stel- 
lated dodecahedron, which turn out to be the first perspective representation of this Keplerian 
solid, as claimed previously. It may be interesting to show the peculiarity of this prodigious solid 
by exploiting the highly communicative and immersive AR experience. In fact, the small Keplerian 
stellated dodecahedron is distinguished from the common elevated dodecahedron thanks to 
the continuity between the sides of the pentagons of the original dodecahedron, with the edg- 
es of the pyramids of the contiguous stars (fig. 2b): a property that can be better understood 
through virtual model that appears in our devices enhanced by perceptual information.The real— 
time interaction with the 3D solid emphasizes its peculiar geometric properties. Thanks to this 
observation, the iconographic meanings of this innovative scientific presence can be illustrated 
in a simple way in a seventeenth-century fresco of a lady's bedroom, whose beauty is honored 
thanks to the golden ratio harmonies of the geometric solid [4]. The comparison with its illus- 
trious predecessors is of interest too. So, we can show the orthogonal representation of the 
solid appeared in a mosaic of the basilica of San Marco, traditionally attributed to Paolo Uccello 
(around 1425) (fig. 2c). We can also cite the illustration by Leonardo da Vinci, contained in De 
divina proporzione by Luca Pacioli (1509) which represented a generic elevated dodecahedron 
with reduced projections and the consequent lack of continuity described (fig. 2d), which was 
also found in the perspective illustration by Daniele Barbaro (The practice of perspective, |569) 
which described an elevated dodecahedron with much taller pyramids (fig. 2e). 

Room of Leda and its features (spatial, geometric, artistic and historical) should be perceived by 
the widest possible audience. The use of augmented reality allows us to make them also accessi- 
ble to people with disabilities. The use of an easy-to-handle device is preferable by people with 
mobility impaired. The possibility of using transcriptions and audio descriptions regarding the 
main characteristics can help to enhance accessibility for people with hearing and visual disabil- 
ities. It is therefore an intervention that implements inclusivity, addressing different age groups 
and people with cognitive disabilities too. AR technologies experience is seamlessly interwoven 
with the physical world such that it is perceived as an immersive aspect of the real environment 
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full extent and not competing the frescoes, helping in clarifying some artifices of the past. 


Notes 


[I] The paper was conceived and elaborated as a team-work: the paragraph Architecture and polyhedra in the Room of Leda 
was written by C. Candito, The reconstruction of illusory space by A. Quartara, and Augmented Reality Applications by A. Meloni. 


[2] Analysis from | to 4 were realized by Cristina Candito, with the collaboration of llenio Celoria for the point |; operation 
5 by Andrea Quartara and 6 by Alessandro. Meloni. 


[3] For the identification of the polyhedron and the artistic and scientific influences on Seghizzi's work, cf. Candito, Celoria [in press]. 


[4] For meanings related to golden ratio perfection and hypotheses about their implications, cf. Candito, Celoria [in press]. 
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Abstract 


Competitions for ideas for architecture have always been sources of stimulus and research, paving the 
way for innovative projects that have also guaranteed an economic flow and a naturally consequent 
social response. 

The municipality of Rome, years ago, was the architect of a campaign of this type: the Cento Piazze 
project included, for the first time, a number of competitions and projects that had never been ad- 
dressed by a national and international administration. 

Our intervention aims to reflect on how and to what extent the Public Administration can now 
benefit from new technologies and digital tools for the analysis and survey of architecture and pub- 
lic space. These innovations, in fact, now make it possible to provide design with a valid help in the 
preparation of competitions that increasingly concern urban regeneration at an international level. 


Keywords 
public administration, survey, cento piazze, heritage digitization, planning. 


The Centopiazze Program 


In 1996, the Centopiazze program in Rome launched the national competi 
ith the design of 19 


hood squares, through wh 


ich young professionals were invited to deal w 


tion The neighbor- 


open spaces in Rome. Centopiazze was a unique urban redevelopment program of its kind 


in Rome, through which 


throughout the territory, from the center to the periphery. With the counci 


it was intended to deal with the issue of public 


space extensively 
led by the Mayor 


Francesco Rutelli, and on the proposal of Prof. Arch. Francesco Ghio, a special program coor- 


dination office was set up 


180 public spaces in the ci 
The Neighborhood squares com 


was launched. It is very inte 


compare the ways in which, 25 years ago, the areas of intervention were p 


to today. The graphic and te 


resting, to date, to look at the competition dossier p 


chnical documentation present 


possibilities and means of the time: first of all, a technical plan ofthe intervention 


within the key-plan of the 


through which the different lots involved in the competiti 


in the Mayor's Cabinet. Over the course of about a decade, around 
ty have been transformed, redeveloped or built from scratch. 
petition was the first action through which the Centopiazze program 


recisely in order to 


resented with respect 
in the competition dossier reflected the 


area, with the area 


city of Rome; a brief description of the area with an indication of the 
functions to be established, location, morphological characteristics of the area, road system, green 
system, urban context, historical analysis, urban destinations (with reference to the PRG in force 
in 1996) and the further planned projects in that area; some exemplary axonometric diagrams 


ion were compared, drawn with wire; 


finally, in the most fortunate cases, an aerial photo of the area and some photographs of the site. 


Reinventing Cities. Historical-Documentary Research and Surveys 


Today, after more than 20 


regeneration and does so through the Reinven 


years, the city of Rome confronted again with the theme of urban 
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ting Cities international competition, A global 


competition for innovative, resilient and zero—emission urban projects. |2 cities on 4 continents have 


so far taken part in the competition. 
cipates with 4 different areas that will be subject to redevelopment: 
Ex Mira Lanza in the Ostiense-Marconi area; Ex-Filanda in the San Giovanni area, close to the 


Among these, Rome parti 


Aurelian walls; Ex Mercato di Torre Spaccata, Vertunni, in Torre Spaccata area, on the eastern 
outskirts of the city; and finally the area of the Roma Tuscolana station. 
has decided to entrust the preparation of historical and analysis doc- 
uments and the survey of the three areas currently subject to competition as 


The municipality of Rome 
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ment. The Roma Tre architecture department was awarded the assignment 


responsibility of Professor Maria Grazia Cianci, thus starti 


an external assign- 
with the scientific 


ng the historical documentary research. 


For the realization of the three competition dossiers for Ex Filanda, Ex Mercato di Torre Spaccata 
and Ex Mira Lanza, the working group then dealt with the historical archival research and the 
elaboration of the integrated survey model, and also with the writing of the texts and graphic 
design. The dossiers open with a timeline that traces the main stages and transformations of each 
building, and a chronology drawn up in a discursive, more in-depth and explanatory form. The 
issue concern the urban setting of the project sites, and an analysis of 


first two chapters of each 


the main historical cartography fo 


and reasoned through the 


area where the Ex-Filanda 


lay of the land beyond the 


more technical and zenith, clarifies the territo 


+ 


r understanding the territorial and urban context, described 


accompanying text. The maps were chosen and placed side by side, 
suggesting a critical comparison between the different drawings, as in the case 


of the study of the 


building stands. The topography of the area, leaning against the Aure- 
lian Walls, is compared in the two maps of Du Perac and Bufalini: the first, perspective, shows the 
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walls, in contrast with the dense urban intramural 


fabric; the second, 


rial conformation with the vallis celimontana in 


which we see the acquae vocatur marianae flow, the so—called Marrana di San Giovanni. 


The dossiers contain all the 


graphic archive documentation that 
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resulted from the historical research: 


drawings, projects, surveys, contracts, and historical photographs that are significant for the exhaustive 
description of each building, tracing the history of each of its transformation. Many sources and ar 


chives were consulted, including the Capitoline Hi 


of Heritage of the Municipal 


storical Archives, the Archives of the Conservatory 


ity of Rome, the Archives of the Istituto Luce, the Central State Archives. 
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Fig. |. Ex Mercato di Torre 
Spaccata, side elevation: 
overlaying of the original 
project drawing on the 
orthophotoplan from 
point cloud. 
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Since these are archival documents that are also very dated, each drawing has been the 
subject of a careful cleaning and graphic post-production which has been aimed at making 
the original signs as legible and clear as in the case of the projects attached to the tender 
specifications for the construction of the pavilion of the Ex Filanda in the Castrense valley: 
original documents from 1920 preserved in the contract fund of the Capitoline Historical 
Archive. Based on the results produced by archival research, the various types of documents 
were presented within the files in order to offer critical and reasoned insights, as in the 
case of historical photos. In the case of the Ex Mercato di Torre Spaccata, for example, the 
current state of the building was described through historical photographs resulting from 
archival research, highlighting the transformations that have taken place over time. 

The laser scanner survey represented a separate part of the assignment, as the processing 
of the point cloud of each of the areas was requested. To obtain it, it was necessary to 
integrate two instruments: laser scanner and drone. The two point clouds obtained were 
geo-referenced and superimposed. 

It was decided to present the original technical drawings of the project by offering a critical 
reading, comparing them with the contemporary technical data represented by the ortho- 
photoplanes that resulted from the processing of the point cloud obtained by laser scanner 
and drone. In this comparison made on the façade of the Ex-Mercato on via Settimio de 
Vico, for example, we observe the homogeneous character of the external walls character 
ized by the stepped profile that follows the profile of the internally arranged ‘mushrooms’. 
The comparison between the orthophoto and the relief drawing of the 1980s shows the 
perimeter closure above the market boundary wall. 

Even in the case of the former spinning mill, the same method of analysis was applied: the 
orthophoto of the elevation on the internal courtyard, for example, was compared with 
the original project drawing, highlighting the changes that had taken place. It is possible to 
note, in fact, the construction dating back to 1965 of the accessory bodies that lean against 
the facade, built once the building was managed by the Servizio Giardini of Rome, and which 
still houses the offices. In this sense we intended to give scientific value to the work: not a 
collection of documents, but an integrated re-reading and re-elaboration through modern 
instruments and their conscious use (fig. |). 
Another important fact, in the case of the Ex Filanda, concerns the historical reconstruction 
that was offered through typological analogy. In fact, this building, built in 1920-1921 by the 
Municipal Administration with funds from the Roman Committee of Civil Organization for 
the educational, moral and sanitary assistance of minors living in degraded situations (later 
nown as Ifantiae Salus), it was not possible to find specific archival photographic documen- 
tation. However, since this is one of the three Infantiae Salus pavilions built in Rome in the 
1920s, it was considered useful to refer to and insert in the file the documentation relating 
to the other two buildings: that of Trastevere and that of Testaccio. 
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The Digitization of Archival Documentation and Built Heritage 


One of the main problems that public administrations in Italy are facing is the digitization of the 
entire archival apparatus, not only linked to bureaucratic aspects, but above all land registry and 
concessions and licenses. Considering that the European directives require the use of BIM in public 
procurement (from 2025 also for minimum contracts of € 100,000), the administrations are al- 
ready in strong delay in the digitization of the built heritage which is part of the future programmatic 
planning of the master plans. The three case studies object of the Reinventing Cities campaign have 
for years (or decades) been strategic sites for the municipality of Rome, the subject of the study of 
projects for urban redevelopment and regeneration and subject to a succession of events that have 
greatly entangled the bureaucratic skein made up of restrictions, prohibitions and permits. 

n this background, the urgency required by the call for the execution of the three surveys 
with laser scanner technology to be provided to the participants is no coincidence. This con- 
cern has necessarily imposed a rational and programmatic planning of the survey campaign, 
taking into account the potential of the tools and the due overlap from two reciprocal laser 
scanning stations. At the same time, the authorizations were requested for drone flights that 
were necessary for the photographic campaign and the creation of the discrete 3d model in 
SFM of the roofs (absent in the ground positions). 

The point clouds, with a detail of a point every + Imm, were mutually aligned in ReCap and 
subsequently georeferenced, as specifically requested by the PA, thanks to the markers posi- 
tioned at ground level and for which GPS data has been recorded. The point cloud created 
for each project site does not, however, exhaust all the drawings needs of a design team. In 
fact, although it allows the navigability of the site digitally its interrogation on the formal and 
dimensional aspects or the perception of the neighboring context, the point cloud must be 
imported into the CAD environment to provide two-dimensional drawings in orthogonal 
projections through its management with cutting plans and the redrawing of plans, elevations 
and sections. This operation is essential to provide the basic iconographic apparatus for the 
canonical drafting of an architectural project (fig. 2). 

Not all PAs or even architectural firms know how to manage point clouds whose management, 
reading and interpretation requires experience first of all in the science of drawing, of its discret- 
ized representation, in different scales, of reality, be it real or virtual. In addition, knowledge, how- 
ever technical, of the appropriate software is needed to obtain the desired results. The research 
group engaged in the work of historical research, survey and representation of the competition 
sites, had to produce plans, elevations and architectural sections, as well as orthophotoplanes use- 
ful for understanding the state of decay of the places. Furthermore, the parametric three-dimen- 
sional model was created, starting from the point clouds, from two-dimensional CAD and having 
a constant control with archival documentation, in regard and verification of all design phases. 
The work carried out, in record time, was certainly a first positive experience for the Munic- 
ipality of Rome, a springboard that highlighted many issues of significant importance, first and 
foremost the need for a digital database of the architectural heritage that allow to accelerate 
the bureaucratic and planning processes of public procurement. 


Conclusions 


While it was once the practice to provide two-dimensional CAD, today technology has al- 
owed administrations to add a third dimension and present instrumental surveys performed 
with laser scanners and photo—modeling carried out with the flight of a drone as basic ma- 
terial. The point cloud thus obtained will be the three-dimensional basis for design studies, 
inserting two non-negligible factors into the process. First of all, the now inevitable need to 
survey, survey and catalog the archaeological, historical and contemporary architectural her- 
itage through tools that are not usual for public administrations and consequently the need 
to make use of professional technicians in the sector 

n our opinion, moreover, the digital processing of the surveys required by the PA as basic 
documentation of the insolvency procedures, can represent an intermediate phase towards 


what could become a real virtualization of the inspection and survey by the designers. The 
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translation of the point cloud and the three-dimensional model into an augmented reality ex- 
perience could pave the way towards the possibility, for professionals from all over the world, 
to experience the architectures and urban spaces that are the subject of the competition, 
without the need to visit the project site directly. Following the limitation of the movements 
that we are experiencing due to the pandemic, in fact, the importance that digital can have in 
providing information in real time from any and all parts of the world has been highlighted. 
Just in the last year the need for digital adaptation has been strongly felt in many different 
sectors and research has found new life for projects that were once perhaps only on a hypo- 
thetical basis, but which today are starting to have concrete foundations. 

One can think of the ever wider opening of digital archives of artistic works or archival doc- 
umentation, the concrete creation of apps and navigation sites by museums, the possibility 
of being able to walk and interact within the virtual museum space, thanks to virtual reality, 
perhaps in an even more intimate and intense way than analogue reality. 

n this sense, starting from the experience of digital relief and restitution developed within 
the Reinventing Cities competition, our reflection invites us to make augmented reality an un- 
precedented opportunity to expand borders and shorten distances in order to gain knowl- 
edge, study and transformation of places, even the most distant ones. 
Point cloud or 3D NURBS model, linked by a close relationship of geometric dimensional dependence 
of the second with the first, are both navigable and interrogable digital environments, but they do not 
allow us to have the immersive perception of three-dimensional space as we would have in the real site. 
Although the first step taken with the Reinventing Cities experience is a huge step towards the digitization 
of the historical-architectural heritage, it is desirable that public administrations also proceed towards 
Virtual Reality through the inclusion of digitized models in platforms such as Unreal Engine or Revit VR, to 
ensure immersive design for engineering and architecture firms from all over the world. 
The question raises many other questions that research will have to answer over time: which 3D 
model will be most useful to navigate? How to make the perceptual and sensory aspects as real as 
possible for a more conscious design? How to solve the problems associated with all kinds of 3D? 
The point cloud, although dense, is not material and by approaching the architectures, the points 
and distances between them are identified, allowing a chromatic perception but not of the material. 
The NURBS model, derived from the cloud, through the still inevitable passage of 2d interpretation 
on CAD, is not texturized, and therefore, although making the perception spatial and concrete, it 
does not give us back the chromatic aspects.A mathematical mesh model that can derive from the 
point cloud, can only partially solve the problems related to perceptual aspects, but the historically 
treated issues of the discretization of the points and the inevitable loss of detail, especially of the 
ornamental apparatus, remain. lt is necessary that, over time, the public administration aligns itself 
with the aspects of communication and representation that begin to be typical of museums and 
archives, realizing that its historical — building heritage (of all eras) is in fact an asset which must be 
shared in the appropriate manner and for different types of users. 
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Abstract 


Resuming the cognitive process started in a previous study of integrated survey and interpretation 
of the remains of the Terme del Foro of Cumae in the Parco Archeologico dei Campi Flegrei, with 
this contribution, we intend to set up, a methodology to deepen the investigation already carried out 
on the current consistencies of the thermal system that, through the preparation of a suitable neural 
network managed by Al processes and structured for the systematic analysis of differences, is able to 
formulate reconstructive hypotheses with scientific authority. 

By integrating the possibilities of traditional methods of investigation on architectural and archaeolog- 
ical heritage, it will be possible not only to highlight what can be directly observed in situ, but also to 
enhance traces that are not immediately decodifiable and even latent, automatically emerging as the 
results of comparative investigations on large amounts of data whose logical structure will be clearly 
traceable along artificial neural ‘meshes’ similar to human cognitive processes. 
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Methodological Reflections 

With reference to the cognitive process initiated in a previous study that, starting from 
integrated survey operations, has produced interpretative results for the archaeological si 
of the Terme del Foro of Cumae [Florio et al., 2020], it was intended to identify a research 
methodology that, activating processes of Artificial Intelligence (Al), makes it possible to 
found reconstructive hypotheses for the Cumaean thermal system on the basis of scientifi- 
cally validated processes oriented to a deeper and shared knowledge. The aim is to obtain, 
through the image-based systems and digital visualization technologies, an expanded ‘cul- 
tural accessibility’ for our architectural and archaeological heritage, which allows not only 
to visit the remains — in situ and/or remotely — and perceive the original consistency with 
the help of Virtual Reality (VR) and Augmented Reality (AR), already adopted in various 
important interventions of valorization of sites and museums, but above all to document 
and communicate in a transparent and accessible way the iter and the ‘intellectual integrity’ 
of the research process, allowing users and other scholars to retrace and critically evalu- 
ate, contextually to the story of the stratifications in the various phases of life of the sites, 
the methodological criteria that have guided the reconstructive hypothesis. The use of 
nformation Communication Technologies (ICT) — and the consequent, increasingly refined, 
possibility of creating forms of digital memory capable of associating a very high density of 
data to the assets of interest (directly retrievable data overlaid on previous documentation) 
— is now fundamental in the field of virtual archaeology. Applying to these large masses of 
information the comparative processes typical of Al, one obtains results whose generative 
process, clearly traceable along ‘meshes’ of artificial neural networks similar to human cog- 
nitive processes, is accessible and therefore open to refinement or possible corrections. 
These logical structures, grafting a propulsive field of support for established interpretive tools, 
end themselves formidably to the dialectic of continuous informational feed—back necessary 
for a sustainable predictive formulation of the original sites. By abstracting the multiplicity 
and continuity of reality, they integrate the methods of traditional architectural-archaeological 
documentation, allowing to manage and monitor, through representation, not only the virtual 
reconstructions (and with them the surviving consistencies in situ) but the same process of 
convergence that recomposes the ancient remains, thus bringing scientific authority to digital 
visualizations, constitutively exposed to the risk of overestimation (unconditional appeal of 
contemporary media) if not even vacuity for lack of transparency or hyperrealism. 


The Study Site and the Neural Networks 


The valuable remains of the Terme del Foro were brought to light during an excavation 
campaign in 1952 at the ancient city of Cuma. The study site is located in the so-called 
‘città bassa', “a quarter of Samnite and Roman times that was developed in the flat area 
that stretches to the east between the acropolis and the slopes that rise towards the Arco 
Felice” [Maiuri 1934, p. 143].With respect to the original configuration that rose to north of 
the Temple Capitolium of the Roman city, the Maiuri found in 1958 the rests “of a grandiose 
thermal building of the | and Il century of the empire with vast environments in which the 
rooms of a calidarium with the apparatus of the prefurnio are recognizable’ [lvi, p. 144]. 

Among the most indicative factors to guide the dating of the first plant of the Terme del 
Foro of Cumae archaeologists have identified the correspondence of the planimetric type 
of the spa complex with the so-called half-axial ring type, widely found in Roman sites of 
the Hadrianic period. It is clearly legible in situ, in the semi-symmetrical articulation of the 
environments evidenced by a main scenographic axis which, from the entrance vestibule, 
crosses the intercolumn of the remains (bases and part of the shafts) of two monumental 
columns in marble extending towards the concavity of the semicircular pool of the frigidar- 
ium and by a parallel secondary axis along which the concatenation of the warm spaces 
(tepidarium, sudatorium, calidarium, praefurnia) is arranged (fig. |). 
In support of this hypothesis for the dating is the evidence, in the surviving structures, 
of masonry and flooring techniques, and of materials, typical of the Phlegrean territorial 
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Fig. |. Orthographic 
horizontal section from 
TLS and orthographic 
horizontal projection 
from SAPR of the 
thermal complex with 
indication of the original 
functional destination of 
the various rooms. 


1 praefurnia 2 calidarium 3 sudatorium 4-5 tepidarium 6 descritarium 7 vestibolo 8-9 apoditerium 10 frigidarium 11 cisterı 


context in the above-mentioned period: at a first reading we find with prevalence the 
opus latericium in bessales for the internal curtains of the warm rooms and the opus re- 
tculatum with latericium reinforcements for the external curtains and for the interiors of 
the remaining rooms, and also flooring remains in tessellated with a white background. 
With the aim of intercepting the construction phases of the thermal site to recompose 
reliably the original consistencies of buildings and wall fragments, integrating the results 
of archaeological studies conducted, the in-depth study — which is still in progress, at the 
moment involving the preliminary phase of the research and preceding what will be the 
application and implementation phase — is aimed at tracing in the remains of the ancient 
complex, which are considerably lacunose and compromised (also by having been brought 
to light through a simple excavation operation rather than through stratigraphic excavation), 
those variations and anomalies that can suggest the stratified structures in the various ages. 
Through the predisposition of a neural network managed by Al processes and structured 
for the systematic analysis of the differences — dimensional and density differences of ash- 
lars and mortars, discontinuity of material and facing, texture anomalies (masonry or floor 
tesserae), etc. — in the subsequent research development phase, it will be possible not only 
to highlight what is directly related to the stratified structures, but also to the anomalies 
of the stratified structures. — it will be possible not only to highlight what can be directly 
observed in situ but, by widening in a formidable way the possibilities of traditional methods 
of investigation and observing the considerable practical advantages offered by the speed 
of comparison and deduction of the new Al systems, to systematize and enhance aspects 
and properties not immediately decodifiable and traces even the most latent, such as those 
material and/or color, automatically emerging as the results of ordered comparative investi- 
gations, bringing scientific basis to the hypothesis on the chronological order, up to recom- 
pose reliably, and according to various degrees of plausibility because of the convergence of 
results, the original configurations of the artifacts. 


Studies for the Cognitive Analysis of the Terme del Foro 


Confirming the indispensability of a hermeneutic action at the base of the hypothesis of 
virtual reconstruction of the sites, the innovative applications of Al project the methods and 
the consolidated practices of investigation in the field of ICT and of the complex analysis 
of large amounts of data, where the experience and the methodological rigor of traditional 
research are capitalized in favor not only of a high efficiency and speed of results — the auto- 
matic recognition of apparently insignificant elements, such as fragments or minimal traces of 
events, becomes an extraordinary tool to reveal the dating and recomposition — but above 
all the possibility of making the ‘story’ and the logical structure of the research accessible 
simply through the reading (graphical) of the system of relations established by the neural 
networks. Already in the field of the consolidated methods, the interpretative accuracy of 
the stratigraphic method is precious for the knowledge of the sites: integrating itself to the 
phase of excavation, the method contemplates the drawing up of plans of interface for ev- 
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Fig, 2. Exemplificative 
framework of the 
variations and/or 
anomalies able to suggest 
the stratified structures in 
the various epochs. 


ery recognizable stratigraphic unit (overlays), deducing stratigraphic diagrams (matrix) that 
condense icastically the data found. Other analytical methods, applied in diachronic phase 
with respect to the excavation, have been using for years deductive processes based on 
logics structured by input data, ‘weighted’ relations, levels of investigation, predictive output. 
Through the mensiochronology, moreover, the multiple input parameters do not acquire 
meaning in themselves: size and shape of the tesserae or ashlars, their number per unit area, 
irregularity factors, etc., are considered contextually, as parts of a comparative structure able 
to develop trend lines for the attribution of chronological bands to the buildings of antiquity. 
Artificial intelligence algorithms, genetically inclined to the investigation of complexity, allow 
us to greatly implement these deductive processes, programming operations (of analysis 
and synthesis) capable of investigating millions of data in all their informative potential (big 
data systems) in order to have new answers to questions about the past. The cognitive anal- 
ysis on the Terme del Foro will have to be set up by providing the system with the highest 
number of data — from those that can be taken in situ, by means of TLS and photogram- 


metry (orthophotos, photographs, photomosaics, photogrammetries) to those that can 
be deduced from previous studies or documentation (Roman building systems and those 
of the following ages, documentation related to wall textures, to the materials used, and 
building appropriate image-based algorithms capable of recognizing the different periods 
and reaching, on the basis of thousands of comparison operations (identification of differ 
ences, and similarities, between a large number of images, photographs, drawings, historical 
data) to possible original configurations (fig. 2). If, for example,the experiment were to be 
applied to the current configuration of the wall textures of the calidarium — the last in the 
sequence of the hot rooms and one of the most affected by wall anomalies and by inter 
ventions of reconstruction and/or transformation — the data to be entered would concern 
the reticulatum and the latericium of the mixed work detectable in the external curtain, the 
differences between the pieces analyzed by type of ashlars (cubilia, bessales), their ratio with 
respect to the average size (7x7 cm, for cubilia; from 23 to 28 cm of side, for the bessales), 
the variations in the composition of the mortar (with or without volcanic inclusions), the 
chromatic tonalities of the bricks and their deviation from the uniform orange-pink color, 
etc. It will also be necessary to document and insert images of the inserts in opus vittata, of 
the mixed opus vittata and brickwork, of the brickwork compensations, of different color 
and workmanship, probably relative to a phase subsequent to the first installation and to 
obtain data on the cores between the two curtains (prevalently in opus caementicium), 
investigating the summit wall crests opened by the gradual and progressive deterioration of 
the thermal remains. Above all, it will be necessary to design the neural network, setting the 
layers (depth of the network), the number of different features that the network examines 
at each level (width of the layer), the connections between the data and the layers (weight 
of the relationships) and opting for a ‘convolutional’ structure (same pattern of connections 
between layers) rather than a ‘recurrent’ one (possibility of ‘feedback’ between non-adja- 
cent layers) [Hartnett 2019] (fig. 3). 
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Fig, 3. Simulative 
hypothesis of the 
structure of a 
neural network for 
the extraction of 


discriminating features for 
the original configuration 


of masonry in opus 
reticulatum. The process 


assigns differential weights 


to the connections, 
distinguishing the 
differential relevance 
of the nodes (full and 
‘empty’ nodes) for the 
outcomes; the system 
also provides feedback 
connections, filtering 
out noise and retaining 
only the most relevant 
features. 
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Conclusions 


The results of the elaborative operations activated by the neural network in adoption, as well 
as being archived within the procedural application itself — making permanently available for 
consultation both the data and the results of the research, as well as the related generative and 
cognitive process — may be valuable support for a subsequent three—dimensional modeling 
of the study site. Consistently with the guidelines for a mature virtual archaeology, inspired 
by criteria analogous to the modern principles of architectural-archaeological restoration 
[Brusaporci, Trizio 2013, pp. 55-68], the knowledge of the thermal complex, represented in 
wireframe compositing in order to distinguish the reconstructed parts and allow progressive 
interpretative updates, will be made accessible and permanently monitored by users in situ 
and/or remotely, through the adoption of web-based and open source technologies; more- 
over the applications of AR and the interactive modalities of the digital visualization will be able 
to associate to the 3D model multiple information and to reveal the relationships between the 
existing and the reconstructed, favoring the comprehensibility of the ruins. 
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Abstract 


PD-Invisible aims to enhance the archaeological heritage of Padua hidden by urban development. The 
focus of the process is the creation of an adaptive AR App according to the type of user: professionals 
and researchers on the one hand and cultural tourism on the other The workflow identifies a path 
through the city and connects the artifacts studied by the research; catalogue historical and archival 
documents; detects the structures through laser scanning and photogrammetry technologies; opti- 
mizes the acquired models both from a graphic point of view and through Scan to BIM; develops the 
AR App in Unity 3D.The research offers further insights assuming the use of artificial intelligence for 
this type of applications: differentiate the contents of the App according to the user’s preferences by 
comparing the GPS data with those of the detection devices (camera and Lidar). 
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Introduction 


U 


D-Invisible, an abbreviation that stands for ‘PaDova INnovative VISions — visualiza- 
tions and Imaginings Behind the city Learning’, is a well rounded project that involved 
researchers from multiple fields. Archaeologists, building engineers, architects and in- 
formatic engineers collaborated for one year with the purpose to make visible what 
is, actually, invisible to the eyes of many. The project was funded by means of the Eu- 
ropean Social Fund and the Regione Veneto and included a partnership between the 
Department of Civil, Environmental and Architectural Engineering and the Department 
of Cultural Heritage of the University of Padua. The city has a beautiful yet, for the 
ity, invisible heritage belonging to the Roman age. The urban growth across the 
centuries and, sometimes, also the lack of attention and care, caused its disappearance. 
the aim of the research was to promote and valorize these archaeological remains, 
bringing back their memory and, above all, trying to give them new life. Focus of the 
project was to create a digital environment suitable and available for both an expert 
and non-expert public, that provided the visualisation of the outcomes [Johnston et al. 
2020; Perticarini, Marzocchella 2021]. 
The chosen solution included the implementation of an application for smartphone 
that uses Augmented Reality (AR) and, moreover, Artificial Intelligence (Al). This could 
enable a simplified fruition of contents, diversifying them depending on the user's in- 
tentions and tastes. Multiple data, technical information about the structures, touristic 
paths and similar point of interests, historical narratives together with pictures and doc- 
uments, are able to satisfy a wider public in the perspective of a cultural rediscovery of 
the Roman historical heritage of Padua. The research started with the identification of 
several points of interest across the city, structures connected by a common past and so 
inked with a touristic path. The Arena, near the Scrovegni Chapel, is the starting point 
as it is close to the actual train station, then walking south through the historical centre, 
along the Riviera dei Ponti Romani, there are Altinate and San Lorenzo Bridges, relevant 
Roman structures for trades and the economy of the city, hidden under the surface of 
the street with the tramline. 
Finally, reaching Prato della Valle, we can find the foundation system of the Zairo Theatre, 
also hidden under the water of the canal of this huge and unique square [Bonetto, Pet- 
tend, Veronese 2017]. San Lorenzo Bridge, between these structures, has an excellent 
conservation status and very positive conditions for the exploitation of the selected 
workflow. It is located under the street but enclosed in an underground chamber accessi- 
ble with an old underpass built in the 1950's [Galliazzo 1941; Gasparotto 1951; Carraro et 
al. 2019]. The division in phases of this research project included the initial data acquisition, 
the possibility to achieve an accurate 3D model with texture and information and the 


final AR and Al implementation. 
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The Case Study 


As previously mentioned San Lorenzo Bridge is the case study identified as a starting 
point for developing research in all its phases and as a key to read for further in-depth 
analysis of the topic. The first very important operation that provides awareness and 
understanding of the chosen structure is data acquisition. The acquisition of the prelim- 


+ 


inar knowledge is then carried out, on the one hand through archival and documentary 
research to collect all the sources and testimonies regarding the bridge. 

On the other hand, the survey performed using laser scanning and photogrammetry to 
obtain a virtual clone, geometrically similar These models are the starting point for the 
subsequent modeling phase. Two different approaches were used: BIM modeling with the 
aim of providing a container for the information collected, allowing the exchange of data 
from different sources and the direct connection between different development envi- 
ronments, especially databases. The BIM model was chosen as the invisible backbone of 
data on which the realistic 3d model lean [Bonetto et al. 2019]. 
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Fig. |. Point cloud of San 
Lorenzo Bridge obtained 
from photogrammetry. 


Acquisition of the Invisible Heritages: Archival Memory and 3D Survey for the Digital 
Reconstruction 


Thanks to the permissions given by the following institutions was possible to carry out an 
extensive archival research: State Archives of Padua, San Gaetano — Altinate Cultural Center 
Superintendence of Archeology, Fine Arts and Landscape, Archive of the University of Padua 
and Eremitani Civic Museum. It was possible to collect all the written and graphic documenta- 
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tion, regarding the main archaeological excavations and finds that duringthe centuries allowed 
to add important elements to the study ofthe Roman Age Padua. Alongside, the geometrical 
survey on-field of San Lorenzo Bridge covered the acquisition of two different types of survey, 
one with photogrammetry and the other with laser scanner technology. The Leica ScanSta- 
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tion P20, a time-of-flight scanner, was employed with the support of Leica HDS black/white 
target for the registration. A total of 18 scans were acquired, with resolution settings of 6,3 
mm @ 10 m, and 78 targets were measured. The registration was performed by the software 
Leica Cyclone with a mean absolute error checked on the targets, of 3 mm. 24 vertices were 
measured as reference networks using a total station Leica TCR1201 and thanks to the GNSS 
was possible to obtain the geographic coordinates. Two photogrammetric surveys were also 
carried out: the first obtained by means of a Nicon D610 full frame reflex camera with a 35 
mm lens; the second by means of two 12 mp GoPro Hero 5 and focal length of 3 mm. The 
photographs were taken following an orthogonal grid of 50 x 50 cm for each arch of the 
bridge and the use of GoPro was useful in reaching the narrowest and most inaccessible areas. 
The point cloud generated by photogrammetry was compared with that obtained by laser 
scanning and no particular differences were found (fig. |). 


3D Modelling and Optimization for AR of the Photogrammetric Survey 


For the creation of the App, a precise workflow was identified to correctly display the 3D 
models on mobile devices. The first step consists in cleaning the point clouds and creating 
complex meshes with a high level of detail (the point cloud, the complex mesh and the 
related texture were obtained with the Agisoft Metashape software). The second phase 
consists in the optimization of the complex mesh by means of retopology (process that al- 
lows to drastically decrease the number of polygons of the mesh, making it lighter and more 
manageable on all smartphone models) [Palestini, Basso 2017] and by means of the baking 
of the normal map and diffuse map (process that allows to imprint the level of detail of the 
surfaces on the texture). For this purpose, the open source Blender 3D software was used 
for baking operations and the Instant Meshes tool for retopology operations [Perticarini et 
al. 2020]. Nowadays, with the latest software releases, two retopology systems have been 
implemented within the software: one based on the Quadriflow algorithm and one based 
on OpenVDB (Voxel). The third phase involves exporting the models to Unity software 
and developing the app in AR using Vuforia and Google's ARCore SDK. The App consists of 
a Ul interface that allows navigation within the map and the identification of artifacts along 
the route; each artifact is visible both on site (through augmented reality) and remotely. It is 
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Fig. 2. above, the complex 
mesh obtained from the 
point cloud; below, the 
simple mesh obtained 
from the retopology 
operation. 


Fig. 3. Ul interface of the 
app. From left: the map, 
the interactive menus, 
the 3D model viewer, 
the glossary and the AR 
interface. 


Glossario 
dei Monu 


also possible to view the historical information and the characteristics of the construction 
elements with a tap on the interactive 3d model, thanks to the implementation of the BIM 
model on a layer below (figs. 2, 3). 


Future Implementations of Artificial Intelligence 


A further development of this type of applications could be the use of Al algorithms, in 
particular machine learning and artificial neural networks, which performs the function of 
satisfying the needs and interests of the user [Jankovic 2020].An example could be to struc- 
ture an algorithm capable of comparing the data coming from the GPS of the smartphone 
— relating to the most visited points of interest or the type of gait — with those coming from 
the camera (the photographs that the user takes most frequently or which artifacts are 
framed instead of others). Data mining consists in identifying information by extrapolating 
from large databases and allows, through associations and recurring patterns, to develop an 
ever—evolving App and an impulse for the user: thanks to the interpolation of data, it offers 
new suggestions based on the preliminary and continuous learning of the user's tastes [Ye, 
Qiu 2021]. One of the most used data mining techniques is the neural networks (ANN 
or NN) adaptive systems that change their structure based on external or internal infor 
mation during the learning phases. The branch of information technology most suitable for 
the information coming from the camera is the automatic recognition and artificial vision 
[Stanisz et al. 202 1]. The computer vision system is composed of typical functions that can 
be differentiated into: image acquisition, pre-processing (resampling, noise reduction, con- 
trast enhancement and space scaling), feature extraction (border lines or ridges ), detection 
or segmentation (small portions of images that are more useful for learning), processing and 
finally the decision-making process. As for image acquisition, current high-end smartphones 
are equipped with increasingly advanced sensors; in particular the iPhone 12 Pro and Pro 
Max are equipped with a Lidar sensor the portable version of the Lidar found in self-driv- 
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ing cars for scanning and recognizing objects along the way [Shih, Diao, Chen 2019] [Yao 
2020]. The sensor could be an excellent tool to improve the algorithm in the classification of 
artifacts framed by the smartphone. For the creation of APP of this type, PyTorch is used: a 
deep learning framework (class of machine learning algorithms) developed by Facebook Al 
Research (FAIR) group. This open source software allows you to facilitate the development 
of algorithms compatible with current Nvidia video cards and take advantage of the GPU. 


Conclusions 


Technological innovations are changing everyday objects into complex intelligent machines. 
The new sensors for acquiring the surrounding environment and the new processors make 
it possible to create projects that were unthinkable until a few years ago. it is obvious that 
often, the use of these technologies has negative sides, especially as regards the use of big 
data and privacy issues. With greater reason, innovations should be exploited to the maxi- 
mum for more noble purposes, such as the enhancement and preservation of historical and 


cultural heritage. 
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Abstract 


This research elaborates a strategy to guide users and scholars in the Byzantine iconographic world, 
highlighting the elements that contribute to recognizing the sacred figures represented [2]. It devel- 
ops two visual approaches: on the one hand, the recognition of faces through a database; on the 
other hand, the use of artificial intelligence for face recognition. The results of the research can be 
applied to the development of content for new media edutainment; for the digital restoration of the 
frescoes; for the communication and enhancement of the asset itself. 


Keywords 
Al, iconography, Byzantine, Saint Nicholas of Myra. 
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Research Protocol 


Byzantine iconography is an extremely complex phenomenon [3] for both the syncretism 
of different cultures and the rigid rules underlying the production of sacred images. The 
historical events that accompanied the expansion of the Byzantine culture in the Mediter- 
ranean area, the repeated Arab invasions, and the transition from the Eastern to the Latin 
rite have compromised the morphology of sacred architecture and their iconographic ap- 
paratus. The rock architectures are directly linked to the migration of monks who, under the 
influence of iconoclastic struggles and repeated Arab invasions, sought shelter by migrating 
from the East, Cappadocia, to the West, southern Italy and eastern Greece (fig. |). 


For obvious reasons, the rock churches are isolated, difficult to reach, and often abandoned. 


® 
D 


riferimenti 

01_ Abbazia di Santa.Croce_Polizzi Generosa_Palermo 

02_ Chiesa dello Spedaff Scalea_Cosenza 

03_ Chiesa di San Zaccaria_Caulonia_Reggio Calabria 

04_ Santa Lucia_Melfi_ Potenza 

05_ Chiesa della Panaghia_ Rossano Calabro_Cosenza 

06_ Santa Lucia alle Malve_Matera_ 

07_ San Nicola dei Greci_ Matera 

08_ Chiesa San Biagio a San Vito dei Normanni_Brindisi 
Fig. 1. Rock churches 09_ Chiesa Santa Croce_ Massafra_Taranto 


15_ Chiesa delle Sante Marina e Cristina_Carpignano_Lecce 
16_ Chiesa di Santa Maria degli Angeli_Poggiardo_Lecce 
17_ San Giorgio_Kurbinovo_Macedonia 

18_ Chiesa Boyana_Sophia 

19_ Chiesa della Koimesis sAsinou_Cipro 

20_ Hagia Sophia _Istanbul 

21_ S. Salvatore in Chora_Istanbul 

22_ Chiesa san Nicola_ Derme_Antalia 


23_ Karanlik Kilise_Goreme_Cappadocia (dark church) 


3 24_ Yilannli Kilise_Cappadocia 
: à 10% Chiesa Madonna della Buona Nuova_ Massafra_Taranto: ‘ 
of the oriental rite. 117 Chiesa di Santa)Barbara_Ginosa_Taranto PENH . 
Localization of surveyed 12_ Chiesa di San Leonardo_Ginosa_Taranto 27--Bahattin Samanli a Gappatiocia 
case studies and formal 13_ Chiesa di San Vito_Gravina in Puglia_Lecce = ae app 
A pic 28_ SumbullusKilise_Cappadocia 
references 14_ Cripta del Crocifisso_Ugento_Lecce 9” Direkli Kilise Cappadocia 
4:21 PM 4:21 PM 57%) 4:21 PM 
= Eastern Church Fa Eastern Church Fathers Byzantine Churches Eastern Church Fathers 
TPHIOPIOS 
OAI FIOPIOC, 
u 
oe 
d Iwávvns 
Fig. 2. St. Nicholas of IE N San Gregorio Nazanzieno 
Nazianzen. Identification 
system based on an recognition faces Gregory Nazianzeno_identikit identikit 


iconographic database. 


The resulting disuse, paradoxically, preserved them by excluding from continuous adjust- 
ments to the changing needs of the cult and society. At the same time, the abandonment, 
together with numerous deliberate damages of the iconoclastic era, compromised the leg- 
ibility of the iconographic apparatus. The proposed system guides users into the Byzantine 
iconographic world, highlighting the elements that help recognize the represented sacred 
figures. This system can be developed in two different ways: by recognizing faces already 
presents in a database or by using artificial intelligence to recognize faces and virtually re- 
storing them (fig. 2). In the first mode, the system identifies the representation of the Saint, 
offering a comparison with other images present in the neighbouring regions and highlight- 
ing recurring elements [4]. Moreover, the system underlines the elements that contribute to 
the identification of the saint himself [Grotowski 2010; Innemee 1992]. 
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The second mode, which uses Al to analyse the morphology of the face of St. Nicholas of 
Myra, is useful for recognition and virtual restoration [Maronidis 2014]. 

Four phases describe the research strategy: |) definition of the survey field; 2) choice of 
the case study; 3) definition of the Al workflow; 4) application of the obtained results to 
recognition and reality enhancement. 
The field of investigation is divided into two parts. In the former the architectures that can 
be part of a circuit of rock churches that are homogeneous in terms of dating, geographica 
ocation, iconographic apparatus and analyses carried out. These churches date between the 
ninth and fourteenth centuries and belong to the ancient Thema of Sikelia, (transformed 
after the Arab conquest into Thema of Calabria) to the Thema of Langobardia (the current 
Puglia), and to the Thema of Hellas, in Thessaly. In the latter part, the need to broaden the 
survey base and give strength and structure to the automated Al process leads us to include 
the iconographic apparatus of many other architectures, which belong to the southern part 
of the Mediterranean, and homogeneous to the first ones for dating [5]. 

The case study is the effigy of St. Nicholas of Myra, widespread throughout the south. The 
effigy of the Saint appears very repetitive and codified. For this reason, this effigy is suitable 
for an analytical investigation that starts from some preliminary morphological hypotheses. 


Fig. 3. Al, input data of the 
reference models. 


Al Workflow 


The artificial-intelligence-based technique used to analyse the effigies of Saint Nicholas 
has been implemented in Python and exploits the Scikit-learn library [https://scikit-learn. 
org/]. The considered input is the set of 19 images of Saint Nicholas reported in fig. 3. The 
workflow involves three phases. 
|. In the first phase, the model input was generated. To identify the position of each point 
on the face, a reference system with abscissas and ordinates was used that marks the ver- 
tical symmetry axis of the face and centres the distance between the pupils. This distance is 
constant in all images and makes a proportional comparison between the selected images. 
For each image, the positions of 50 specific points of the face have been identified and 
measured in the two—dimensional Cartesian coordinate system. 

Specifically, we show in fig. 3 the points selected in the face are marked by three different 
colours. We mark in blue the points related to the proportions of the face, such as face 
width and height, eye position, nose, and mouth. Green is used for the points related to the 
morphology of the somatic features, such as the shape of eyes, mouth, and beard. Finally, the 
points that cannot be detected on a single face are marked in black. 

2. In the second phase, missing data have been replaced by the mean of the column, and 
obtained data have been normalized into the range O and |. 

3. In the third phase, we used the k-means clustering method to partition the images into 
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Fig. 4. Al clustering results. 


Fig. 5. Al clustering; 
visualization of the data 
obtained. 


several groups in such a way that images with similar characteristics belong to the same 
group. We varied the number k of clusters, and the results of these analyses are discussed 
in the next section. 


Conclusion 


We performed several analyses varying k (i.e., the number of clusters). For space limitations, 
we report the results of the most interesting experiment with k=3. The three clusters 
show differences in the face size and lip crease and do not seem to identify a partitioning 
related to the painting dating or the geographical location (fig. 4). in a further analysis, we 
overlapped the images in the same cluster and tried to ‘visualize’ the distinctive features of 
the faces. The division into clusters becomes more precise and a differentiation emerges: 
the reduced size of the forehead (cluster 2) or the size of the beard and chin and the more 
pronounced fold of the lip downwards (cluster |). Cluster O shows intermediate features. 
The experimentation is only at the beginning, and we expect that with the introduction 
of other data and the collaboration of an Iconographer, it will also be possible to obtain a 
reliable hypothesis for the reconstruction of the missing parts (fig. 5). 

The model obtained with Al made it is possible to identify some preliminary strategies for 
the morphological classification of the iconography of St. Nicholas through the identification 
of the complex proportional relationships between the parts of the face. The results of the 
recognition and cataloguing carried out by artificial intelligence can be applied to: creation 
of implementable databases for the dissemination and documentation of the Byzantine 
iconographic heritage; development of content for new media related to edutainment; 
digital restoration of the asset; communication of the asset aimed at the enhancement and 
consequent conservation of the asset itself. 


cappadocia |costantinopoli | rus ala ia sud italia 


russia ortodossa 


cluster 0 
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Notes 


[1] The research presented in this paper is the result of the joint work of the two authors. The Research Protocol is created 
by Marinella Arena, the Al Workflow is in the name of Gianluca Lax, the Conclusion is jointly signed. 


[2] IThis study is part of a wider research, between lonio and Egeo conducted with Daniele Colistra and Domenico Mediati, 
which has surveyed many sacred buildings of the Byzantine era. 


[3] “at the actual stadium of our knowledge we are not able to establish links between the saved pieces of art and the social 
structure or the byzantine geography (..) some general facts reaffirm this pessimistic attitude: the systemic anonymity of the 
byzantine pieces of art, the extreme lack of the written sources, caused by the disruption of almost all the archives; in the end 
he insufficient differentiation between the pieces of art.” [Grabar 1964, pp. 27-28]. 


[4] Just as in language few signs, combined differently, generate a multitude of words, similarly in Byzantine sacred representations 
he sacred vestments, the nimbus, the homophorus, the polistavrion, or the receding hairline, the colour of the hair or the 
shape of the beard, identify the Saint's figure. Cfr “Pursuing the ‘representation’ rather than the ‘imitation’, Byzantine artists used 
a limited range of forms. Within these stylistic norms they changed the physical appearance by modelling the shape of the 
head and the outline of the cheeks. They used lighter or darker pigments to achieve different skin tones. A wider colour and 
variety of form was possible in depicting hair and beard. They could be dark, red or white, long or short, whereas the absence 
of beard and moustache was meant to suggest young age. This method allowed the creation of a very limited number of 
face-types. This is confirmed by monotonous descriptions in iconographic manuals and eikonismos collections. * [Grotowski 
2010, pp. 137-138]. 


[5]. Geographical and chronological location of the case studies: |_Chiesa del Crocifisso Lentini, Siracusa, 1000; 2_Chiesa 
dello Spedale, Scalea, Cosenza, 1000; 2b_Chiesa dello Spedale, Scalea Cosenza, 1000; 6 a_ San Nicola Mottola, Taranto, | 100; 
6 b _San Nicola Mottola, Taranto, | 300; 6 c _ San Nicola, Mottola, Taranto, 1300; 7 _ San Nicola dei Greci, Matera, | 200; 8_ 
San Lorenzo, Fasano, Brindisi, 1200; | 3_ San Vito Gravina, Lecce, 1300; 14_ Cripta del Crocifisso, Ugento, Lecce, 1350; 15_ SS. 
Marina e Cristina, Carpignano, Lecce, 959; | 6_ S. Maria degli Angeli, Poggiardo, Lecce, | 200; 18_ Chiesa Boyana, Sophia, | 250; 
22_ San Nicola, Derme, Antalia, 800; 30_San Giorgio, Sophia, | 250; 31 _ San Nicola, Steyis, Cypro, | 100; 32_ San Nicola Bolnicki, 
Macedonia | 350; 34_ Monastero Eski Gumus, Turchia 1000; 35_ S. Marina, Muro Leccese, Lecce, 1087 
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Abstract 


The development of ICT has favoured the spread of real-time, pervasive and ubiquitous applications. In 
particular, VR and AR visualizations allow a close interrelation between people, data, environments and 
objects. Consequently, it is possible to enrich cultural heritage with information by visually superimposing 
multimedia content, in absolute respect of their physical consistency. In this way, it is possible to create a 
‘smart heritage’ dimension that combines the potential of the ‘Phygital’ with the protection and enhance- 
ment of assets often characterized by important elements of fragility. An important role is played by Al 
applications, which automatically direct the processes of ‘Interpretation’ and ‘Presentation’ of the heritage. 
Based on the experience of the 3D reconstruction of the no longer existing Baroque configuration of 
the Basilica of Collemaggio in L'Aquila, aim of the paper is a theoretical-methodological reflection on the 
concept of VR / AR / MR for cultural heritage. 
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Fig. |. Views of the point 
cloud of the Basilica of 
Collemaggio. 


Introduction 


The development of ICT has favoured the spread of real-time, pervasive and ubiquitous 
applications, which allow the interrelation between information, people and environments. 
In this way, thanks to smart devices, each user is constantly immersed in an on-life state 
[Floridi 2015]. In this context, Artificial Intelligence (Al), combining data, algorithms and 
computational power through machine learning processes, plays a very important role. In 
particular the White Paper on Artificial Intelligence: a European approach to excellence and 
trust [European Commission 2020] highlights the potential of Al for citizens, economic ac- 
tivities and the public good: “the impact of Al systems should be considered not only from 
an individual perspective, but also from the perspective of society as a whole. The use of Al 
systems can have a significant role in achieving the Sustainable Development Goals, and in 
supporting the democratic process and social rights [...] Promoting the adoption of Al by 
the public sector [...for an...] An ecosystem of excellence”. In the field of cultural heritage, 
Al can significantly influence the processes of interpretation and presentation underlying the 
conservation and enhancement of cultural heritage [Tielden 1957; ICOMOS 2008], playing 
a leading role through user profiling with repercussions in the methods of storytelling and, 
above all, the mechanisms of interaction and participation. The interaction between virtual 
reality, augmented reality, mixed reality and artificial intelligence can lead to direct conse- 
quences in the context of the methods of defining and declining experiences in the context 
of digital heritage [Brusaporci 2017; Pierdicca et al. 2020]. 

The paper presents a theoretical-methodological reflection on the concept of smart herit- 
age starting from VR/AR/MR visualizations for cultural heritage, which is based on an expe- 
rience related to the 3D digital reconstruction of the no longer existing baroque configura- 
tion of the Basilica of Collemaggio in L'Aquila (figs. |, 2) [Brusaporci et al. 2021] [ 1]. 


Smart Visualizations 


The traditional static approach to ‘smart cities’, centered on infrastructural networks, has 
developed in the sense of a cultural dimension, where the concept of ‘smart cities’ has 
evolved into that of ‘smart places’, in particular thanks to participatory declinations. Oper 
ationally, ‘smartness’ requires the integration of ‘objects’ with sensors and their enrichment 
with information, so that they can interact with the environment. If the so-called ‘Internet 
of Things’ poses problems of a substantially technological nature if applied to contempo- 
rary products, when it is addressed to artefacts characterized by historical and aesthetic 
values, issues of a theoretical-methodological nature arise. These relate both to the image 
of the work and to the respect for its materiality, testimony to the events and cultures 
that have taken place over time. Therefore, it follows the need for precise reflections on 
the methods that can favour the information enrichment of cultural heritage. The well- 
known principles of documentation, recognisability, reversibility and protection, in particular 
when interacting with the physicality of the assets — for example with sensors or tar 
gets — remain firm. One solution is to interact with the artifact only in the visual field, 
through temporary and mediated views that enrich it with information. This is the case 


Fig. 2. Collemaggio AR 
app for viewing the 
baroque configuration 
that no longer exists. 


of video mapping [Rossi 2013], but above all it is what is offered by augmented reality 
applications [Ch'ng et al. 2017; Clini et al. 2017], able to automatically recognize what is 
framed by the camera of the device and to superimpose information of various kinds. 
This type of experimentation is conducted by the research group of the University of L'Aquila, 
starting from the INCIPICT project [http://incipict.univaq.it/] integrated with the 5G exper 
imentation, through a proprietary app specially developed for the enhancement of cultural, 
architectural and urban heritage of L'Aquila [Brusaporci et al.2019].The application, which can 
be used in urban areas of the city and in specific interiors, is implemented to tell the history 
of monuments and places, also by superimposing in real-time 3D virtual reconstructions of 
past configurations that no longer exist (figs 3, 4). Specifically, the paper presents the expe- 
rience dedicated to the Basilica of Collemaggio, the heart of the rite of ‘Perdonanza’, which 
was included into the UNESCO Intangible Cultural Heritage Lists in 2019. Furthermore, the 
AR app opens to lines of research aimed at Al, with for example image recognition, user 
profiling, prediction and management of contents and interactions [Pierdicca et al. 2020]. 
Conceptually, the close interaction between real heritage — experienced in its own historical 
environment — and digital visualization refers to the concept of ‘Phygital’ [Nolaf 2019] and 
strengthens its applications in the field of cultural heritage. 


The AR App by Univag 


Mobile augmented reality (Mobile AR) is gaining increasing attention from both academia 
and industry. Hardware-based Mobile AR and App-based Mobile AR are the two dominant 
platforms. However, hardware-based Mobile AR implementation is known to be costly and 
lacks flexibility, while the App—based one requires additional downloading and installation in 
advance and is inconvenient for cross-platform deployment. However, with the improved 
communication and computation capabilities provided by 5G technologies, a combination 
of both technologies is growing up in order to support tourists and cultural applications, 
Furthermore, the emergence of 5G mobile communication networks has the potential to en- 
hance the communication efficiency of Mobile AR dense computing in the MEC approach [2]. 
Several technological advances have started to enter the landscape of Mobile AR. First, the 
upcoming 5G networks [Coluccelli et al. 2018] bring new opportunities for Mobile AR. 
They provide higher bandwidth (0.1-1Gb/s) and lower network delay (I1-10Oms), which 
improves the data transmission on mobile networks. Second, the introduction of new char- 
acteristics, such as MEC, device-to-device (D2D) communication, and network slicing, pro- 
vides an adaptive and scalable communication mechanism that further provides efficient 
infrastructures for the deployment and promotion of Mobile AR. 

Within the INCIPICT project a MEC based demonstration testbed has been set-up. The 
system exploits the platform available at the MEC LAB of the University of L'Aquila in order 
to validate the capabilities of the MEC architecture to support applications dedicated to 
AR services. The MEC LAB provides a complete and customizable network environment 
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Fig. 3. Screenshot 

of the Collemaggio 

AR application: 
reconstruction of the 
baroque ceiling and 
superimposition with the 
current configuration. 


Fig. 4. Views of the 

reconstruction of the 
Baroque Collemaggio 
within the application. 


and consists of 3 nodes distributed in the city of Aquila. In the University hub, more than 15 
physical servers are available and interconnected using optical and wireless technologies to 
provide heterogeneous connectivity between nodes up to 10Gbps per network segment. 
The laboratory hosts also a 5G radio access network and a core network to implement 
network slicing with guaranteed performance on a common physical infrastructure and it 
is used to perform edge computing experiments. The availability of distributed computing 
infrastructures in the city allows the experimentation of the orchestration of virtual services 
in metropolitan networks required by the AR services. 

Moreover, a mobile app based on Apple's ARKit has been developed in order to evaluate 
service. In order to exploit the system some 3D models were created in different sizes. AR 
was made by the developed application available for 5G smartphones. All 3D models were 
stored into the MEC platform and accessed by the 5G network minimizing the latency and 
with a very high throughput in order to provide the best user experience. Once available 
within the app a user can browse the 3D model using the device camera. 


Conclusion 


The examples offered by Google and Amazon to all users highlight how mixed reality appli- 
cations are and will be increasingly widespread. Therefore, it is believed that the analysis of 
the possibilities offered in the field of knowledge, conservation and enhancement of cultural 
heritage is of great importance. The experiment presented is only a first step within a line of 
research dedicated to the study of architectural heritage. 

We want to conclude with a general reflection on the digital dimension of the visual in the 
narration of cultural heritage, starting from the well-known “Uncanny Valley” by Masahiro 
Mori, that is, his graphs dedicated to the sense of affinity of people towards robots, in relation 
their resemblance to humans and their ability to move.We could think of updating Mori's re- 
flection by replacing robots with digital visualizations (VR, AR, MR), and the concept of'move- 
ment’ by the influence of machine learning, able to simulate our behaviour in an increasingly 
effective way and to prevent our requests and wishes more and more efficiently. In the current 
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so-called “Age of Culture” [Schafer 2014], ethical questions arise where Human Computer 
Interaction can prefigure new reflections and experiments aimed at the interpretation and 
presentation of cultural heritage. In this context, Al plays a leading role, where users are in- 
creasingly accustomed to the use of synthetic and cunning images in the relationship between 
real and digital [Brusaporci 2019]. 


Notes 


[I] The research has received funding from the Italian Government under Cipe resolution n.135 (Dec. 21, 2012), project 
INnovating City Planning through Information and Communication Technologies (INCIPICT). 


[2] MEC, formerly mobile edge computing, refers to the enabling technologies that provide computing capabilities and service 
environment at the edge of the network [European Telecommunications Standards Institute (ETSI) White Paper]. 
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Advanced Practices of Augmented 
Reality: the Open Air Museum 
Systems for the Valorisation and 
Dissemination of Cultural Heritage 


Gerardo Maria Cennamo 


Abstract 


This paper aims to explore the broad topic concerning the use of virtualization technologies in order 
to earn knowledge of the cultural heritage and it's dissemination. The activities aimed to develop the 
use of these advanced technologies, in the archaeological and architectural heritage promotion, are 
constantly evolving. This is the case for the museum fruition, that was confined to places of conser 
vation and contemplation until a few years ago, but now exportable to whole urban sites (open air 
museums) thanks to the support of the virtuality that introduce to immersive learning paths. Very 
important resources in the cultural heritage valorisation process are the fruition improvement and 
the active involvement of the guest through the use of immersive paths, both pedestrian and vehicular. 


Keywords 
cultural heritage, representation, museum experience, advanced technologies, augmented reality. 
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Fig, |. Open air museum: 
experiment of immersive 
AR path, Naples Rione 
Sanita 


This paper is part of a broader research work, already anticipated in other scientific dis- 
cussions dealing with the use of the reality's virtualization technologies applied to the en- 
hancement of the cultural heritage; this technology represent a powerful tool to improve 
knowledge, learning and dissemination supports, but it only should be considered effective 
when its use is codified through rigorous approaches. 

Strategies for knowledge disseminating through efficient and enjoyable communication 
approaches, include diversified levels of in-depth analysis (such as those offered by ad- 
vanced technologies in the fields of surveying, post-processing and representation). These 
strategies take on an indispensable role as support of the scientific research methodolo- 
gies in the archaeological and architectural heritage activities, sometimes overcoming the 
instrumental value and configuring themselves as an individually recognizable step of the 
research process. 
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The gradual development of the digital virtuality, which began in the 70's of the last centu- 
ry [I], has had a large spread that was proportional to the constant migration of the IOT 
technologies towards easer platforms, widely usable and accessible to most people. This 
defines a widespread acceptance of a new transmission code of informations or a new 
system of symbols, typical of the virtual environment; already considered by the research- 
er M. Forte in 2004 — with some residual doubts — is the exponential development of 
widespread applications to be integrated into exhibition tour for museums, widely usable 
by all the people and not the prerogative only of a specialist users [Forte 2004]. 

The full diffusion of the digital and simulation technologies, now achieved, obliges us 
to share this new informations transmission code, that is based on various cybernetic 
interactions [2] and is alternative to the natural one that is based on known spatial and 
perceptive rules; already at the beginning of this millennium, some researches took into 
consideration this possibility, resulting from the diffusion of virtual reality and its dynamics 
of information exchange, from which it would emerge “un nuovo codice percettivo dello 
spazio e del tempo in cul la prospettiva, insieme alle operazioni mentali di temporalizzazi- 
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one e spazializzazione che essa presupponeva, viene definitivamente messa in discussione 
e, divenuta un'alternativa tra le altre, si de-oggettivizza” [Pecchinenda 2003, p. 49]. 

By circumscribing the argument in the interests of this study, we can be maintain that the 
main purpose of a digital processing is to increase the perception capacity (of the object 
itself as a cultural asset) and it’s resulting semantic charged. In other words, the digital trans- 
position of a cultural asset introduces a complex process whose effects include a extensive 
review of the asset itself for its reinterpretation and dissemination in a virtual way. We can 
assumed that the studying, analysing and processing activities applied to the cultural asset 
itself for the purpose of its transposition into a digital way, can bring out a features (and their 
interpretations) alternatives to those obtainable from the approach not aimed at simulation 
activities in a virtual environment. In this way, a rigorous approach to the cultural heritage 
produce multiple variables of knowlodge: ‘‘Lepistemologia del virtuale suggerisce alcune 
riflessioni circa lo scambio e la geometria di informazioni fra reale e virtuale, fra soggetto e 
oggetto della fruizione culturale nell'ottica di una nuova musealizzazione virtuale di dati ed 
informazioni culturali” [Forte 2004, p. 429]. 

By spreading this considerations, it is clear that the need to code a rigorous syntax not only 
of the digital models (which are the basis of augmented virtual reality) rather of the entire 
procedure of developing and managment in the virtual environment, still finds ample space 
in the scientific debate. 

To date be lack a univocal code, a procedural guidelines to refer to develop these specific ac- 
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tivities. This situation represents an anomaly not only of epistemological value but also norma- 


tive one, that we must be considered as any approach to the cultural heritage is, to date, reg- 
ulated both in terms of methodological aspects and in terms of the definition of the outputs. 


Fig, 2. Open air museum: 
augmented contents 
(Palazzo Sanfelice, Rione 
Sanita). 


The analysis procedures and surveying systems are, for example, rigorously codified, the 
methodologies of approach to study, research and planning recognized and internationally 
shared, the typologies and number of the outputs to be produced are well regulated. Even 
the use of digital in the AEC scope, is harnessed within a regulatory system that encodes 
and regulates the processing phases [3]: the processing of technical and designing data 
into the BIM process, are governed by regulatory and unifying rules issued both by the EU 
and by the nationally jurisdiction [4]. It seems to slipping away a shared regulation, for the 
approach and implementation method, especially in the direction of the archaeological and 
architectural heritage, specific for the virtually tools. 

To better clarify the concept, we should be remember that the virtualization of works art 
or cultural assets in the museum buildings is a well-established practice that finds excellent 
examples; there are many important museum organizations that have elected the virtuality 
as the ordinary modus for the public offer of their heritage. The most usual use, in this case, 
is the virtual reality (VR). On the other hand, the possibility of converting entire areas or 
urban districts of historical and archaeological interest where to develop immersive paths 
of perception through digital reality, makes use of augmented reality (AR) [5]. 

Semantic issue: the activity already described is not free by some aspects that still need to be 
explored. For example, we can considered, in the case of the ‘museum translation’ of entire 
sites through the instruments of the augmented reality, that the semantic weight bring out of 
this experience have a greater concreteness than a similar one lived in a completely virtual 
environment. This because the digital transposition of scientific contents (so-called augmented 
contents) finds a direct and one-to-one correspondence with the real context, developing 
with it continuous interactions and exchanges of information that are not expected due to the 
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multiple variables introduced by the guest himself. Instead, the many museum organizations 
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al, exploratory or perceptive) but in which he can freely exercise arbitrary choices, placing 
himself in an active one-to-one relationship with the virtual environment: “In RV tutte le 
informazioni sono interconnesse in uno spazio 3D; una ontologia della connettività implica 
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rs position along the path, while 


In order to make the synchronism between the approach of the visitor and the delivery of 
multimedia contents reliable, the interaction system is entrusted to particular transmitters 
(beacon) which, positioned near the points of interest, send information managed by the 
application pre-installed on the guest's devices. 


Conclusion 


The studies in order to codifying and to controlling the potential of the reality simulation ap- 
plications about the archaeological and architectural heritage, are constantly evolving, which 
confirming the important opportunity linked to the new frontiers of approach in this spe- 
cific scope. Although in a simulated way, these processes have their main objective in the 
description of the cultural heritage within its site of origin; in addition to the strong contribu- 
tion carrying out from the technologies, we must remember that the effectiveness of these 
systems is based on a methodologically correct approaches in the phases of investigation, 
analysis, surveying and processing, which are typical of the representation sciences. About the 


+ 


identification of precise guidelines, useful to implement the correct use of these technologies 


+ 


in order not to ‘succumb’ to the digital and virtuality potential but, vice versa, to direct their 
contents in the correct direction of a scientific approach, we must to consider the possibility 


+ do 


of defining new syntactic codes within the mechanisms of perception and representation, as 


recently observed by Alberto Olivetti [7]: "Dovremmo riflettere su quanto sia cambiata la 
dimensione della percezione e la consapevolezza diffusa del passato, del presente e del fut 


turo. 
Alludo alla dilagante rapidissima estensione dei mezzi virtuali di tipo informatico che hanno 
profondamente inciso su quel rapporto di spazio e tempo che forma l'endiadi entro la quale 
passato—presente—futuro agiscono poiché il tempo, in una dimensione virtuale, annulla le cat- 
egorie di spazio e tempo quali erano state elaborate solo fino a ieri”. 


Notes 


[I] The first virtual reality system is the one created in 1968 by Sutherland and Sproull, cfr. Biocca, F., Delaney, B. (1995). 
mmersive Virtual Reality Technology, in: Communication in the Age of Virtual Reality, Hillsdale: Lawrence Erlbaum Associates. 

[2] Cfr. https//wwwtreccanit/enciclopedia/cibernetica, definition by Treccani online: Cybernetics, a discipline that focus the 
unitary study of processes concerning ‘communication and control in animals and machines'. It can also be defined as the 


general study of highly organized complex systems, regardless of their nature. 


[3] The acronym AEC (Architecture Engineering Construction) identifies the building sector supported by IT approaches. The 
reference literature is very large and finds important interests in the area of the representation. 


[4] In Italy: the law about BIM is the D.M. 560/17; the regulation about BIM is the UNI | 1337-7. 


[5] A brief definition of augmented reality by Treccani online cfr. https://wwwtreccani.it/encyclopedia/ augmented-reality: 
virtual reality technology (AR) through which a digital contents are added to the real environment. In opposed to the concept 
of virtual reality (VR) which develops fully virtual environments. 


x 


The reference is to conversion of full historical sites in open-air museums. On the subject cfr. Cennamo, G. (2018). 


[7] Cfr. Olivetti, A., Stati Generali della Memoria, Universita Telematica Internazionale UNINETTUNO, Roma, 2020. 
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Abstract 


The course of Performative Techniques in Visual Arts (2020-21 edition) at Libera Accademia d'Arte 
Novalia revolved around a project which saw students actively involved in the promotion of a local 
heritage site: the Precettoria (Abbey) of Sant'Antonio di Ranverso (Province of Turin). Despite its 
cultural relevance this site is little known, although the last few years saw an intense campaign for the 
promotion and restoration of the complex. The project completed during the course is part of this 
effort and the management of the site has been involved in its completion. 


Keywords 
augmented reality, heritage sites, Accademia Novalia, educational services. 
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Goals and Methodology 


Precettoria of Sant'Antonio di Ranverso: 

Among the sites managed by the Fondazione Ordine Mauriziano [|], we decided to focus 
on the Precettoria of Sant'Antonio di Ranverso because this complex, located at the en- 
trance of the Susa Valley [2], is a prestigious site whose late-Gothic architecture and rich 
and wonderful frescoes bring us back in time to the age of the Duchy of Savoy, between the 
dusk of the Middle Ages and the beginning of the Renaissance. 

The social and historical relevance of the complex of Ranverso during the medieval period 
originated in | 188, when the Duke of Savoy offered the site to the Hospital Brothers of 
Saint Anthony. The members of the religious order hosted pilgrims and patients suffering 
from shingles (commonly referred to in Italian as fuoco di Sant’Antonio, meaning ‘Sant An- 
thony's fire’) and, later, from the plague. The Precettoria was conveniently located on the 
Via Francigena, the pilgrim route leading to Rome and some of the foremost religious sites 
of Christianity. Over the centuries, the entire complex has been rebuilt several times, while 
the church was completed during the last three decades of the | 5th century, commissioned 
by Jean de Montchenou, who was named Prior in 1470. Back then, the complex included a 
hospital (of which only the façade is surviving), the church, and the Precettoria building. Of 
particular importance are the frescoes painted inside the church, depicting stories from the 
life of Saint Anthony the Great, the life of the Virgin Mary, the Passion of Christ, and the life 
of Saint Blaise of Sebaste, painted by Giacomo Jaquerio and his atelier in the first quarter of 
the |5th century: they are a true masterpiece of the International Gothic. 

At the end of the 18th century S. Antonio di Ranverso presided over a vast territory and, judging 
from the number of rural buildings, the surrounding area must have been considerably populated. 
By the time of the suppression of the Order of Saint Anthony in 1776, the territory administered 
by Sant'Antonio di Ranverso included almost a quarter of the area of the town of Buttigliera Alta, 
plus four farmsteads: all buildings and the land were assigned by the Pope Pius VI to the Ordine 
Mauriziano, and now belong to the Fondazione Ordine Mauriziano, a foundation who converted 
the site into a museum and is now safeguarding and promoting it. 

Despite its cultural relevance, this site is little known: this is due to the lack of touristic promotion 
(which started only very recently) and the location of the Precettoria, which is currently cut off 
from public transportation. The restoration of the church, carried on between 2015 and 2017, has 
marked an important step: the lighting inside the church has been entirely redesigned, and a ticket 
office and a bookshop were added. 

From the opening of the site to the public in June 201 7, special efforts have been made to promote 
the visibility of the Precettoria, to establish educational services tailored for different visitors (from 
schools to families), and to use social media to reach potential visitors both in Italy and abroad. 

In this framework, the work carried out as part of the Performative Techniques in Visual Arts 
course at the Libera Accademia d’Arte Novalia [3], in collaboration with the site management, 
was marked by the signing of an agreement between the parties involved to grant the stu- 


Fig. |. The site, seen from 
the Via Francigena. 
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dents access to all the available study material and total freedom of action inside the complex. 
The project was carried out in two phases. Phase | was carried out on site, to assess the 
true extent of the tour of the entire complex — and not just of the church: it was immedi- 
ately clear how an easily accessible audio-guide was key to fully understand the area and 
its surrounding, including the exterior of the building facing towards the Via Francigena, next 
to the hospital’s façade. 

Beyond a guided tour led by a museum curator, who is not always available, the only sources 
of information for the visitors were a few info pillars located inside the church, under the 
portico, and inside the cloister. We decided to create educational content and made them 
available to visitors; considering that the site is mainly toured by families with children, we 
aimed at engaging that specific audience. 

Phase 2 was a detailed survey of the needs of the site, of the Accademia didactics, and of 
the users, which we can summarize as follows: 


U 


recettoria of Sant'Antonio di Ranverso: 

*Create an interactive guide available during the tour of the site; 
«Minimize costs; 
Design an object for the museum shop; 

Not having info pillars, panels, or other structures installed on site. 


Accademia didactics: 

e Enhance the students’ skills: the Accademia focuses on art techniques and the history of 
art, but not on coding (any need for that must have been outsourced, reducing the involve- 
ment of students in the project). 

Users [4] 

«Explore the site; 

«Minimize costs; 

Not having apps installed on personal devices (because data plan costs, storage space issues, etc.) 
«Engage children without resorting to tablets or mobile devices. 

The data we collected have been analysed and used to evaluate the potential of differ- 
ent solutions, like info pillars with NFC chips, VR, AR with illustrated markers, AR with the 
mapping of existing elements to be used as markers, an e-book or in-app guide, and AR 
with QR codes. All potential solutions and technologies were tested and analysed by the 
students, as summarized in the following table: 


Guides | Low cost Not having Low cost (for | Children | No info | Shop 
(for the site | apps installed | users) pillars 
management) on persona needed 


Info pillars 
with NFC 
chip 


AR 

with 
illustrated 
markers 


AR 

with 
mapping 
of existing 
elements 


E-book 
or in-app 
guides 


AR with 
QR code 
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Fig. 2.The first map 
selected for testing. 


As clearly shown, the only choice meeting all the criteria proved to be an illustrated map 
with QR codes linking to files from an audio-guide recorded by the students. 

The project was carried out in 5 steps: 

|) Production of all the assets needed to complete the project. First of all, the students 
visited the Precettoria and identified 8 points of interest along the visiting route, based on 
their relevance to the learning process related to the site: the goal was using them to build 
an itinerary both inside and outside the complex, to explain in very simple words the spirit 
of the place, the emotional journey of the pilgrims, the everyday life of the monks, and the 
work of the atelier of a major painter on the | 5th century frescoes. 

2) Writing the audio—guide text and recording it. The language used is precise but acces- 
sible to everyone, including families with children. 

3) Designing the map. We opted for an A4 size prioritizing illustrations, lowering printing 
cost but also making it user-friendly even for children. The map can be folded and held easily 
with one hand while holding a mobile device in the other. 

4) Testing the map.As soon as the layout was ready, the students were asked to print a copy 
of the map and test the QR codes in low light conditions to ensure that they would work 
inside the Precettoria. 

5) Production of the final map file, ready for printing. 


Discussion 


The AR and VR technologies are more and more widespread today, so there was much 
discussion about the opportunity of resorting to a simpler AR technology such as QR codes 
for this project, and if that might have led to a greater risk of obsolescence. The choice of 
QR codes was made to increase the degree of independence for students working on the 
project but also with the visitors of the Precettoria in mind: a comparatively low-tech prod- 
uct is far more suitable for their audience. 


Dago 
Du 


MASSO ERRATICO 


JAQUERIO 


OSPEDALE 
SANT'ANTONIO 


It took almost 10 years for QR codes to become widespread to the point where the vast 
public is now able to recognize them and use them with no need for specific instructions. QR 
codes are now printed on billboards for supermarkets' promotional sale, on school books 
used in primary schools, and even on menus, a growing use seen during the Covid-19 epi- 
demic. Most of the mobile phones today don't need a specific app to read them: they can be 
read using a phone camera. This assessment led to the conclusion that QR codes are familiar 
and easily accessible to a vast and diversified public, while AR markers require a specific app 
and VR is not suitable for children under the age of 12 (and can be unpleasant for many 
adults). QR codes might also eventually be integrated with NFC chips for greater accessibility 
for visually impaired visitors. Furthermore, this technology is so common that the first survey 
about the implementation of QR codes in museums, promoted by the SMartArt [5] project, 
dates back to 2013. We can safely assume that QR codes will be the standard for museums in 
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the near future. The main innovation in this project is where the QR codes are printed: not on 
info pillars or panels but on an illustrated map, a physical although low-cost object made more 
attractive by illustrations. The map is able to draw the attention of the visitors who, when they 
return to their homes, are still able to maintain a dialogue with the site thanks to the audio 
contents, which can be accessed and listened to anytime they want. 

The students were considerably engaged by this project. Opting for a technology such as 
the QR code, so easy to use, didn't affect the visual and creative design but only the poten- 
tial need for coding. The quality of the final works has been consistently high. Unfortunately 
the visitors’ feedback has been delayed by the measures and restrictions in place during the 


current Covid-19 epidemic. 


Conclusions 


This project has proven extremely beneficial both for the didactics and for the Precettoria. 
We hope that the low cost and the relative ease of content production in this format will 
encourage a similar approach in many other heritage sites which, similarly to Sant'Antonio 
di Ranverso, are maybe little known but of great historical and artistic importance: similar 
contents can be implemented where a visiting itinerary is missing or to promote itineraries 
tailored for families or visitors belonging to specific categories. Regular updates will also help 
form a long-lasting relationship between the site and its visitors. 
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Notes 


[I] The site of the Precettoria belongs to FO.M. Fondazione Ordine Mauriziano, a foundation whose aim is to protect and 
promote the Precettoria of Sant'Antonio di Ranverso, the Abbey of Staffarda and the Palazzina di Caccia of Stupinigi, which is 
a UNESCO World Heritage Site. 


[2] In Buttigliera Alta (Province of Turin). 


ces) 


] The Libera Accademia d’Arte Novalia of Alba (https://novaliaarte.com/). 


[4] Users’ needs have been assessed based on the information collected from visitors and by creating fictional ‘Personas’. 


‘a 


] QR-CODES in MUSEUMS, http://www.smart-art.it/qrcodes-museums/ (23 February 2021). 
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Abstract 


The project presented here is addressed to the documentation, the investigation of architectural values 
and their valorization through an application of Augmented Reality technologies enhanced by an Al 
based tracking application of the Sanctuary Basilica Madonna dei Sette Dolori (BVMA: Beata Vergine 
Maria Addolorata) in Pescara. The workflow foresees the use of the numerous images taken for the 
phases of photogrammetric acquisition of the artefact and images taken from the visualizations of the 
cloud of laser scanner points in order to carry out the “education” phase of the Al software (so that 
the program can store the greatest number of images for a self interpretative reconstruction of the 
geometries). The Al data will then be used as a tracking structure for the AR overlay of the digital model 
on real space, all through a webXR application usable from any device (HMD, desktop or mobile). 


Keywords 
segmentation, virtual heritage, machine learning for heritage, augmented reality for heritage, deep learning. 
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Introduction 


The project is to be included in the context of cultural heritage enhancement practices, through 
the use of digital technologies of three-dimensional modelling, Augmented Reality and Artificial 
Intelligence. The aim of the project is the development of an AR navigation device that allows 
the interactive visualisation of the pre-conciliar configuration of the church in real time. The 
contribution of digitisation in enhancing heritage and supporting our awareness of our history is 
significant (Cameron, Kenderine 2007; Pavlidis et al. 2007; Pescarin 2016; Luigini 2019), and AR 
technologies add the plus-value of a natural fruition, compared to what happens with VR. 

The Sacrosanctum Concilium of 4 December 1963, drafted within the Second Vatican Council 
held between 1962 and 1965, introduced important innovations concerning the Catholic 
liturgy and liturgical space that started an important season of adaptation of the existing 
churches, especially in the area of the presbytery. 

The XIX National Eucharistic Congress held in Pescara in 1977 was the opportunity for 
the upgrading of the Church of BVMA, and the main interventions were: the removal of 
the balustrade that bounded the presbytery, the replacement of the altar with a frontal altar 
and the removal of the ambo at the height of the first span, with the consequent place- 
ment of the current ambo on the presbytery. The artistic value of the new artefacts does 
not coincide with those replaced, and so the project to make the early twentieth-century 
configuration usable again is motivated by the need to restore an architecturally significant 
configuration to the church. 
The church, built in several stages from the second half of the seventeenth century until the 
mid-nineteenth century with the construction of the bell tower; is in the shape of a Latin cross 
with three naves covered by elliptical low domes, and a tripartite facade with a pediment in 


the centre supported by two pairs of pilasters and two side wings with another two pairs 
of pilasters. The digital reconstruction work will concern, in particular the apse area and the 
facade, the latter plastered in the mid-twentieth century and recently exposed (fig. 1). 


Fig, |. Lateral section of 
the church obtained from 
the point cloud of the 
laser scanner survey. 


Artificial Logic Construction: State of the Art 


Al allows computers to imitate human cognitive processes in order to configure a logic in 
which ‘learning’ and ‘solving problems’ automatically brings considerable benefits in many 
application areas, where its consistent use can be found in Computer Vision, such as Ren- 
dering, Facial Recognition, Video Post Production, but also in 3D survey procedures, laser 
scanning or photogrammetry, applicable in disciplines such as Geomatics and Architecture: 
multidisciplinary studies, both theoretical and practical, are progressively opening towards 
spheres aimed at the analysis and dissemination of Heritage, clearly involving the transver- 
sal use of new media, such as AR. New ML technologies and AR dynamics could improve 
the development of applications that can be exploited in education or Heritage enhance- 
ment projects, amplifying space/user interactivity. In the future, a progressive development 
of smart applications is therefore envisaged, combining the new technologies of AR, DL 
and Semantic web focusing on the recognition of objects in different conditions, even very 
complicated ones, retrieving relevant information through semantically linked open data 
and interactively augmenting this information in real time in a real perceptual environment 
[Lampropoulos et al. 2020]. 
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Fig, 2. Complete workflow 
of the search process: 
from document search to 
AR application. 


AR and Al effectively cooperate together, making them the two most promising technolo- 
gies available to mobile application developers without the need for complex programming 
steps, since machine learning models use self-generated data from which patterns and 
correlations are learned, and AR, capable of merging physical environments with digital 
content. Thanks to the integration of the two technologies, the credible superimposition of 
digital elements on physical objects also makes any element captured, subject to any kind of 
investigation, questionable through an accurate digital segmentation self-recognised by the 
computer This opens up exceptional potential uses by providing new ways of interacting 
with the physical and digital worlds. The calculations in question refer to the Deep Neural 
Network, the term ‘Deep’ expresses the function in which multitudes of data—layers, similar 
to the human neural system, transform the input data to generate solutions by means of 
repeated processes of automatic compilation. In short, it is a form of automatic learning that 
learns to recognise the real world by identifying it through nested hierarchies of visual sam- 
ples in which each individual conceptual model is defined as the result of other abstractions. 
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The Application Case Study 


This research aims to employ the open source tools available on the web for AR technolo- 
gy with the integration of some algorithms that allow editing of Al-based space3D tracking: 
a polygonal model of the church of BVMA in Pescara, reconfigured from a previous laser 
scanner survey according to the latest sculpting modelling techniques, is visually superim- 
posed through an App for smartphones that exploits the most popular AR systems with 
the support of artificial intelligence according to the preconfigured FRITZ_AI model, to 
offer a tour that can show, with simplicity and immediacy, the architectural evolution of 
the church assuming also the previous historical variants (fig. 4). The workflow (fig. 2) can 
therefore be divided into main phases when the ML procedure employed foresaw the use 
of numerous images taken from the photogrammetric acquisition and images taken from 
the post visualisations of the cloud of laser scanner points. 

The Al data was then used as a tracking structure for the AR overlay of the digital model 
on real space, all via mobile devices. 

In order to allow in AR visualization a credible correspondence of the model, we opted 
for a polygonal modeling that exactly traces the point cloud. An HD polygonal model was 
obtained thanks to the use of the open source software Cloud Compare, through the re- 
constructive algorithm of the Poisson Surface Reconstruction plug-in , which generated a 
mesh capable of credibly tracing, based on the parameters entered, the shape that the point 
cloud assumes from the laser scanner survey starting from its maximum density. In order to 
obtain a versatile polygonal model suitable for an easy transposition on popular virtual plat- 
forms that support AR visualization, it is necessary to optimize the mesh in high resolution 
deriving from the survey by means of auto-retopology and to generate the UVMap useful 
to support the textured mappings that replicate the hyper—detailed model, which currently 
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Fig, 3. Synthetic diagram 
of the functioning of 
GANs, used for tracking, 
and CNNs, used for 
semantic segmentation. 
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consists of a poly count too high to be managed on mobile real time rendering platforms. 
The operations are carried out with Zbrush, an artistic sculpting program often used in the 
workflow for the 3D management of hyper-realistic architectural assets [Trizio et al. 2019]. 
Machine Learning can currently be considered an indispensable tool for improving AR apps. 
The case study used the 'FRITZ-AI_ML Platform’ , an online resource that, by provid- 
ing Al-based ‘training’ models, allows developers to use image datasets immediately for 
production without the need to compile any code. Both Core ML and TensorFlow Lite 
models are automatically generated, making it easier to develop apps with ARKIT4 and 
ARCore functionality. Through a univocal interface and a progressive and guided workflow, 
it is possible to choose the type of model you want to use right from the start, based on 
the functions supporting the App, including Object Detection, which identifies objects in 
an image and draws a bounding box around them to make them interrogable, Pose Esti- 
mation, which predicts the position of specific key points in an image to perform precise 
tracking; Image Segmentation, which enables automatic recognition of framed objects using 
pixels; or Model Labeling, which can recognise people, places and things based on an ML 
model trained on millions of previously ‘labelled’ images. For the generation of any model, 
however, it is necessary to load a Data Set of images functional to the Al training. There 
are several types of external collections currently in use, such as Oasis Dataset, but for an 
optimal result the best solution is always to generate, as input of the model to be built, a 
custom dataset according to the supported mobile ML Frameworks. Thanks to the Dataset 
Generator function, FRITZ Al offers the opportunity, starting from a few sample figures, 
to automatically generate a dataset consisting of numerous artificial images with elements 
ready for intelligent segmentation. In the case study in particular, about twenty sample 
images taken from the laser scanner survey and deduced from photorealistic renderings 
carried out specifically by external applications were used. Approximately 500 images were 
returned, starting from 20 input images, which were then used, in order to configure a 
ML/Tensorflow Core Model, in the Al training based on the recognition of the Corinthian 
capital (applying the Object Reconition Model by means of the input of various images 
of capitals acquired photographically inside the church) and subsequently on the specific 


recognition of some key pixels to carry out tracking in the AR environment (applying the 
Pose Estimation algorithmic Model). In this way it was possible to generate a self-compiled 
Open Source code that could be subsequently used in the Android framework archi- 
tectures, in the development of the App currently under completion. The Vuforia system 
compatible code, thanks to which it is possible to use many of the most common AR func- 
tionalities, will allow us in a future phase to link the virtual model to some elements of the 
frame, in our case decorative elements of extreme recognisability and repetition such as 
the baroque capitals of the pilasters. The structure will therefore support, especially in the 
wide framing, where there are more points—target capitals, a solid base to support a stable 
visualization. As mentioned, it will also be possible to attribute to the elements framed by 
the cam additional contents, historical data, photo galleries, further model-elements that 
can be manipulated in 3D. These tools essentially offer the possibility of reducing produc- 


tion times by increasing the quality and consequently the interactivity with the user a ploy 
that in short supports engagement policies for the participation of Cultural Heritage. 


Conclusion 


Following the digital survey phase, we reconstructed three—dimensionally the interior of the 
church hall and the exterior, with particular attention to the reconstruction of the pre-con- 
ciliar configuration. The following workflow was dedicated to the development of the AR 
device, able to visualize in a natural way the previous configuration of the church. This device 
will be used to allow the fruition of the architectural and artistic value of the church, in the 
most significant configuration of which there is evidence, with the aim of preserving the 
aesthetic and architectural qualities of the artefact and to spread greater awareness among 
the public of the values that the local heritage has expressed, also in view of the lack of pro- 
tection. The awareness—ralsing programme will include an online communication campaign 
and activities with local schools, to foster interest in visiting and learning about the heritage. 
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Abstract 


The research investigates the theme of the valorization of the huge, but widespread, archaeological 
heritage of the Phlegraean Fields which, already weakened in its conservation and fruition by the 
bradyseismic phenomena of the area, is made even more fragile by the absence of narrative strategies, 
making even local communities unable to perceive its value. The study proposes a systematization of 
the knowledge of the Phlegraean Fields Park, through surveys and 3D models, integrated by the use 
of different digital technologies, which together promote effective forms of communication between 
users and heritage. Each site becomes the node of a network of thematic routes, traced starting from 
the major attractions of the area and aimed at defining a hybrid landscape, made of in site visits and 
immersive digital experiences. The goal is to generate a new model of inclusive museum, configuring 
cultural relationships between physically distant places, between lost spaces and real ruins. 
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Introduction 


The unique landscape of the Phlegraean Fields, as a palimpsest rewritten over the centuries 
by complex phenomena of volcanic nature, boasted in the years of the Grand Tour an undis- 
puted fame among European travelers. They recounted in numerous paintings and engrav- 
ings the wonder and fascination of the ancient classical ruins of the Roman era immersed 
in a suggestive natural context. Since the 20th century, however, the link between nature 
and archaeological evidence has been abruptly altered by the unplanned expansion of the 
modern city, dominated by an uncontested and widespread building abuse. The close, and 
sometimes inseparable, connection between archaeological sites and modern construction 
[Di Liello 2005] has strongly influenced the methods of preservation and enhancement of 
the Greek-Roman remains. The Phlegraean Fields Archaeological Park, in fact, is a fragment- 
ed complex, consisting of twenty-five archaeological sites located even several kilometers 
apart. The Park has many problems, including the state of abandonment of large parts of 
the heritage, the lack of services, access and transport networks, as well as the inadequate 
participation of private individuals in the cycle of conservation, enhancement and manage- 
ment of cultural heritage. The impossibility of expropriating private buildings, moreover, does 
not allow the highlighting of archaeological assets, often even hidden by private individuals 
themselves. These critical points do not allow to enhance the heritage according to the 
most modern and shared strategies of conservation and musealization, nor to consistently 
organize the system of services for accessibility and presentation of architectural findings to 
the public. The aim of reconnecting the Phlegraean archaeological heritage encourages the 
search for a new communication strategy capable of integrating all the sites in the identity 
of a single large widespread park that, overcoming the physical fragmentation of today's 
urban fabric, can recompose the original and unitary territorial system of the Roman period. 


The Phigital Archeology Project 


The aim of returning the areas affected by the archaeological excavations to the life of the 
contemporary city, giving dignity and value to the ancient remains, has guided the research 
towards the use of appropriate digital communication technologies. These technologies 
not only allow to replace the physical visit where impractical for structural or security 
problems, but also to build new forms of relationship between citizens and the ancient ur- 
ban fabric. ICT and digital networks increase, in fact, our ability to access information and, 
therefore, knowledge. The design of an integrated exhibition, partly physical and partly 
digital, made of real movements and virtual paths, physical spaces implemented and digital 
immersions, also allows to overcome the fragmentation of the Phlegraean archaeological 
heritage, creating new, more active and emotional ways of narration and fruition. The first 
step was the construction of a transversal corridor between places because “When we 
experience territories, we create stories. We model these stories using mental maps” 
[laconesi, Persico 2017, p. 277]. The creation of thematic maps, explorable and question- 
able, and narrative paths allows to connect archaeological sites even very different and 
distant, but linked by a common identity matrix. It involves placing certain sites in a the- 
matic transect [Diedric, Lee, Braae 2014], which creates connections even where they are 
no longer visible. The routes of visit and knowledge, organized according to the original 
use of the sites and included in a special interactive map in Google Mymaps, are: Theaters, 
Amphitheaters and Stadiums; Water Sites; Temples; Burial Sites (fig. |). Each path, involving 
a large site attractor, could characterize the monthly tourist offer of the Park: in this way 
the minor sites could benefit from a flow of visitors not easily recallable, thus justifying 
the costs of the opening of some, otherwise visitable only on request.A process of digiti- 
zation of the built heritage was then started, through a scientific collaboration agreement 
with the Park, using photogrammetric Structure-from-motion (SfM) survey techniques, 
which could return 3D mesh models with high definition textures. These models allow to 
reconstruct digitally a faithful hypothesis of the original configuration of the good, which 
becomes a tool of great effectiveness for the communication of the ancient value of the 
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Fig. |. The four 
thematic routes for the 
Archaeological Park 
of Phlegraean Fields: 
Temples; Water places; 
Burial places; Theaters, 
amphitheaters and 
stadiums. 


monument. Despite the presence of numerous historical and architectural studies, in fact, 
the understanding of the archaeological vestiges continues to be difficult for the general 
public: the loss of the major volumes, coatings and colors, compromises the possibility of 
appreciating the heritage. The digital reconstruction of the original state, as well as the 
relocation of sculptural decorations lost or removed for protection needs, would allow 
the people of the Phlegraean municipalities, first of all, to suggestively enrich the emo- 
tional impact in situ, ensuring not only a deeper and more conscious path of knowledge, 
but also the definition of new relationships between the contemporary city and the an- 
cient urban fabric. The digital models, moreover, constitute the indispensable basis for the 
technological tools with which we want to implement the narrative. The contextual use 
of augmented and virtual reality has been added to the more basic use of QR-Codes 
to link via web to multimedia content. The project also includes the physical installation 
of descriptive and graphic panels, which are intended to develop a new form of direct 


interaction between users and heritage. 


E 


> 
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Analogical and Digital Augmented Reality 


Augmented Reality is one of the choosen strategies, put in practice thanks to the open 
source app Augment since it allows you to overlay a new layer on archeological remains, 
the one of digital reconstruction of the structure. A 3D model of the real architecture is 
shaped through a philologically reconstruction, based on the literary sources and on sim- 
ilarities of coeval and affine buildings. Through the correct detection of an insertion point 
into the 3D model, the digital content becomes automatically visible in the camera while 
looking at a marker, that is associated with the 3D model's link while designing. This allows 
to project, in the same frame, the reconstruction of monument directly onto the arche- 
ological remains, completing them if necessary because they are incomplete, absent or 
unrecognizable due to the time. For example, the “Sepolcro di Agrippina”, so called because 
of a wrong denomination during the XVI century, shows the ruins of an ancient theatre of 
Giulio-Claudia period. The loss of the top floor, cavea and stage that were the most char 
acterizing elements of the typical theatral roman architecture, changes the original shape 
making it unrecognizable: the chance of exploring interactively the morphology and the 
spaces of a three dimensional model, created on an affordable reconstructing hypothe- 
sis, becomes a successful way to communicate even without specialized spectators, with 
difficulties in spatial imagination (fig. 2). This way, the user can experience the aspect ratio 
of the architecture reconstructed in its original shapes compared to his physical position, 


143 


Fig. 2. Digital model of the 
‘Sepolcro di Agrippina’ 
inserted in the Augment 
app for Augmented 
Reality (frame the marker 
with the scan of the app 
and the 3d model will 
appear as in the photo). 


Fig. 3. Example of 
analogical augmented 
reality. 


changing the framing with the only obligation of sighting the marker printed on the panel. 
This way we provide a hybrid and multimodal experience where, the personal perception, 
physical and essential, of the visit to the archeological site, becomes an interactive tour. So 
the knowledge process is supported and implemented by the experience through informa- 
tion, spaces and digital objects integrated, in a mixed reality, to real ones. The design has a 
conceptual graphic style, with a simple monochromatic texture, that associates to the digital 
model the meaning of “drawing of real”. Such choice allows the visitor being aware he's 
looking a likely reconstruction that doesn't excludes the further configurations as well as he 
can differ from the excessive hyperrealism of some augmented experiences that, alming to 
sensationalism, make the observer a passive viewer rather than a visitor In these case the 
archeological approach is drawn and influenced by technology [Volpe 2013]. The same goal 
is provided also thought the set-up of transparent panels that offer, for each archeological 
site, one or more perspective images, properly taken from the three dimensional recon- 
structed environment, reproducing the direct view that the observer would have of the ar 
cheological building in a particular position of the expositive path. The finding of the correct 
relation between archeological fragment and digital reconstruction is given to the visitor 
that reassembles the view by overlaying three “red spots”, existing in the drawing on the 
transparent panel, to the three corresponding markers applied on the equivalent points of 
the real physical structure. In this case we can talk of “analogical augmented reality” because 
this strategy is characterized by a real space augmented with new signs and high interactivity 
actively and emotionally involving the visitor to recognizing the lost parts related to the real 
ruins. The augmented reality, both analogical and digital, gives the chance to overcome the 
dichotomy between the physical and digital space (fig. 3). However, when some spaces are 
not accessible any more, it has been integrated with virtual reality experiences, ensuring the 
sensation of a physical experience inside buildings now impenetrable. 
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Conclusions 


The proposed “Phygitalarcheology” project is an integrated set-up, partially physical and 
partially digital that allow to provide new kind of enjoyment, hybrid and multimodal, of ar- 
cheological sites, ensuring new spatial relations among sites physically far each other, among 
lost spaces and real ruins, real and digital spaces. This integrated process generates a new 
model of museum, more inclusive, where digital information is not referred and attached 
only to the single object or site, but recombine, remix and recontextualize themselves cre- 
ating always new physical and semantic geographies. The direct and fundamental experience 
of visiting the site, implemented by digital contents, becomes therefore a narrative-interac- 
tive path, encouraging not only the reconnection of the heritage diffused on the Phlegrean 
area, but even a new sense of knowledge of its value that would reconnect the citizens to 
their archeological heritage. 
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A Technique to Measure the Spatial 
Quality of Slow Routes in Fragile 
Territories Using Image Segmentation 


Andrea Rolando 
Domenico D’Uva 
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Abstract 


The current research aims at investigating the potential of image segmentation (IS) technology, based 
on web application, for measuring the spatial quality of slow routes. The big amount of street-level 
images, publicly available through several applications such as Mapillary, Google Street View, are rele- 
vant sources of information, that allow virtual explorations of many places around the world. The (IS) 
technology allows partitioning of a single digital image into sets of pixels in order to read and recognize 
the visual content within the frame of the image. By applying IS technology to the images taken along 
a defined route, it has established a method for grouping images in relation to their spatial features. 
The method has been applied to some stretches of slow-mobility routes, that are localized along the 
fragile coastal landscape of Trabucchi, south of Italy. A selection of images along the route, both in the 
outdoor and urban space, has been analyzed, with the aim to test the effectiveness of the method, able 
to produce useful information to define a Spatial Quality Index. 
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slow routes, mapping, spatial quality, image segmentation, fragile territories. 
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Introduction 


The methodology presented in this paper results from a search for a solution to the 
problem of assessing the spatial quality of slow mobility routes [Scandiffio 2019; Bianchi 
et al. 2020], that are located off the beaten tracks, crossing territories that are marginal 
and distant from densely populated areas. The majority of studies on the subject refer to 
major urban centers, neglecting fragile areas of the territory. These areas are often char- 
acterized by a structural lack of data, which makes in-depth analysis difficult. However, it 
is possible to perform advanced research methods in the field of spatial analysis, by ex- 
ploiting the potential of street-level imagery, users’ generated contents that are available 
on the web portals. These methods belong to the Artificial Intelligence (Al) family. It is 
beyond the scope of this work to make a detailed analysis because, within this particular 
type of software, there are many subsets, each of which has a specific function [Zhang et 
al. 2018; Zhang et al. 2019; Cao et al. 2018]. The current research refers to a particular 
system of Machine Learning, named Deep Learning, that is based on Neural Networks 
[Buratti et al. 2020]. This particular subsystem allows analysing imagery and recognizing 
elements already present in an internal database of the machine. The use of an Al system 
is necessary because the database does not contain precisely the identified element, but 
only a series of similar elements that are analyzed and compared until an ideal candidate 
for the final recognition is found. The system, therefore, learns from the elements supplied 
to it and increases its knowledge step by step. Once the element has been recognized in 
the image, an even more specific procedure is used, called Image Segmentation (IS), which 
perimeters the object providing its position and area. Thus, this digital ecosystem returns 
the position and quantification of an element within a single image, which corresponds 
to a geo-referenced coordinate of the landscape. By performing the IS method to a se- 
quence of geo-referenced imagery, it is possible to capture the landscape features along 
the route , which can be used for measuring its spatial quality. It is important to underline 
that the interpretation and assessment of the quality cannot be fully automated in the first 
instance: correct reading and interpretation of images must be prepared, defined and pro- 
gressively tuned through an iterative process of machine training, possibly implemented 
and improved by massive users contributions. The process is in fact based on comparative 
assessments and still requires, in any case, a direct experience of places. 


Case Study 


The case study to which the methodology was applied is the Costa dei Trabucchi greenway 
in Abruzzo Region; a greenway converted into a tourist cycle route since 2017 [Marcarini, 
Rovelli 2018]. This route will stretch, upon completion, for 40 kilometers between Orto- 
na and San Salvo. At the time of writing this work, it is almost wholly completed in two 
sections; the first to the north between Ortona and Fossacesia and the second to the 
south between Casalbordino and San Salvo. The northern section of this route was pri- 
marily constructed by the Adriatic railway line's retreat, leaving a void along the coast. It is 
currently almost completed and is accessible except for the tunnels, which are still under 
renovation. The southernmost section is a route that stitches together different territorial 
environments, the coastal towns of Casalbordino, Vasto, San Salvo, and the Regional Nature 
Reserve of Punta Aderci. This section has been entirely completed, except for the complete- 
y inaccessible tunnels. The coastline landscape along the route is very scenic, with wide and 
sandy beaches that alternate to natural cliffs that are interrupted by small urban settlements. 
Both sections are located protected natural areas of great value such as the Reserve of the 
Butterfly Cave and the Lecceta di Torino di Sangro and Trabucchi in San Vito, Fossacesia, 
and Vasto. For the purposes of the study, shots were selected in 6 sections belonging to 
the two sections. Often, the elements, that are in-between the cycle path and the sea, such 
as vegetation and houses, impede the landscape's unobstructed view, sometimes trapping 
the route in canyons in which the perception of the valuable elements of the landscape is 
seriously compromised. 
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Methodology 


The current methodology starts from the imagery surveying of the route, which has 
allowed to record the GPS track and to take photographs, orthogonal to the route 
direction, every 10 meters, by using the web application Mapillary [Porziy et al. 2020]. It is 
a web platform that automates mapping tools, by collecting street-level imagery taken by 
users with a standard smartphone. Mapillary exploits computer vision tools [Warburgy 
et al. 2020] and it allows the recognition system's application. The methodology is based 
on the object detection in images, based on IS technology. This technology is part of 
the digital ecosystem of Machine Learning that allows the recognition of the perimeter 
of objects and their measurements within the whole image. The objects are assimilated 
to the elements included in a training model already existing in the Machine Learning 
engine, and a percentage of occupation of the framed field of view is provided.An ad hoc 
algorithm has been written in Python language with the aim of extracting quantitative data 
related to each image. This algorithm makes it possible to shift the IS’ heavy computational 
burden from local machines to remote cloud systems that operate the recognition in a 
much more efficient way. The image elements are extracted in the form of very rich JSON 
archives because they include the type of object, the area occupied in the image by each 
object, the recognition reliability percentage, and the coordinates of all the points that 
define the perimeter of the object. The algorithm skims all objects that occupy an area of 
less than 5% of the entire field of view, returning only the family and sub-family to which 
the object belongs, and the area occupied. 


Image segmentation methodology for automatic recognition of landscape elements 


Python 


i Vegetazione (30%) ! 
L A 
n y y 1 


Photographic survey with smartphone | | a 
Fig. |. Workflow of the ne 
Image segmentation 
methodolgy for automatic 
recognition of landscape : : ! 
elements Track definition and data acquiring : Data computing on Mapillary server : Element recognition : Element classification 


The grouping of images has been done by considering three main layouts as reference 
for the scenes. The layouts have been drawn, by selecting the horizontal surface of the 
path, the vertical obstructions and the openness of the sky. By assuming the horizontal 
surface of the path as an invariant component of all images, different thresholds of the area 
percentage of each component of the scene have been applied, in order to analyze the 
surfaces in the surrounding of the path, which, effectively, affect the landscape perception. 
Three different scene-types have been defined for grouping images. The scene-type | 
corresponds to an open environment, with small vertical elements on the sides and on the 
background of the image. The scene-type 2 corresponds to an environment where a wider 
surface of vertical elements is present only on one side. The scene-type 3 corresponds to 
a closer environment, surrounded by vertical elements on both sides. For each scene-type 
two different images, belonging to the outdoor and to the urban environment, have been 
selected, in order to test the effectiveness of the method. Both images of each scene-type 
have been selected in the same range of area percentage values (fig. 3). For each image it 
has been extracted the area percentage of each object in the image, through the Python 
algorithm. The raw values of area percentage, derived from the IS, have been grouped into 
eight main categories (path, sky, vegetation, edgings, water, terrain, buildings, mobile objects), 
in order to simplify the categorization and allow an immediate reading, For each category, 
it has been computed the area percentage. 
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Fig, 2. Comparative table 
of landscape features 
along the "via verde della 
Costa dei Trabucchi” 
greenway. 


Outcomes 


By processing the images with IS, three different thresholds have been identified for assessing the 
spatial quality of the landscape crossed by the route. The threshold values of the sky and vertical 
elements (vegetation, buildings, edgings) seem to be the most interesting indicators to outline the 
features of the scene. 

The images in the scene-type | have the area percentage of the sky more than 40%, and the 
sum of the area percentage of vertical surfaces (edgings, buildings, vegetation and mobile objects: 
people and cars) is less than 30%. The images in the scene-type 2, the area percentage of the sky 
less than 40%, and the sum of the area percentage of vertical surfaces is among 30% and 40%. 
The images in scene-type n. 3 have the area percentage of the sky less than 20%, while the sum 
of the area percentage of vertical surfaces is more than 40%. The thresholds values identified in 
this research, could be modified and adapted, if they are tested in other environmental contexts. 


SCENE TYPE 1 OUTDOOR 1 URBAN 1 


SCENE TYPE 2 OUTDOOR 2 
SCENE TYPE 3 OUTDOOR 3 


MI Path 12% 
y BE Sky 17% 
‘Vegetation 
| Edgings 2% 


Conclusion and Future Research Developments 


The limits of this methodology are twofold.The first is related to the system's problems, the second 
to the limitations of images data provision. The Mapillary system only allows the recognition of 
elements that can be read by referring to a database which is pre-defined by its developers and 
not open to changes. Therefore, some elements typical of the area covered by the case study are 
not taken into account For example, the trabucchi, amphibious machines that are the main heritage 
— from which the coast itself takes its name — are selected as Void objects, not recognized. These 
objects along with other notable features, are not recognized because the Mapillary database only 
includes standard features. It would be necessary to customize the database to select only those 
objects that are expected to be retrieved according to their relevance for our specific qualitative 
analysis. Such an update of the tool could take advantage of the peculiar characteristic of Machine 
Learning systems, namely the ability to learn through analysis, and improve its precision and 
reliability. However the problem arises when training this new database, whose operation requires 
significant machine time resources and the need to rely on external structures to complete the 
process. During the analysis of the case studio, problems were encountered in the recognition 
system because only 100 items per image are analyzed at any one time. The main features could 
be included in this value, but some attempts at element recognition only found 51% of them. 
This limitation can be overcome by directing the system to search for single categories of objects, 
instead of the totality, by expanding the number of detectable elements. To overcome this first 
type of limitations, the evolution of methodology will consist of updating the Python algorithm of 
parameter extraction, e.g, implementing the grouping of the elements recognized in the indicated 
categories. This procedure would start from the sum of the elements with equal features to 
automatically group and draw a graph of what is currently done manually. 

The second type of limitation concerns how photographs are taken.A positive aspect of Mapillary 
is the possibility to survey using a standard smartphone. This flexibility implies a significant problem 
for recognition: namely the angle between the direction (perpendicular along the lens axis to the 
smartphone sensor) and the horizon. Tilting the field of view downwards increases the amount 
of land (increasing the number of elements included and incurring in the problem told above) in 
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the frame. Tilting it in the opposite direction increases the percentage of the sky that fills the frame. 
The incidence of this factor will be the subject of future research developments. A further 
limitation of this type is the standardization of the photos because they are taken with various 
instruments with different resolutions and fields of view, with intuitively different results. The 
differences induced by these limitations can be overcome by using standardized tools for 
capturing photos, implementing a camera whose inclination, position, and shock-absorber can 
be controlled within specific well-defined parameters.A completely different approach to this 
issue Is relying on more reliable photo databases such as Google Street View [Zhanga et al. 
2020; Anguelov et al. 2020] but with limited coverage of the slow mobility routes covered in this 
work. A better definition of the thresholds for each category can be found, in order to apply 
the method to other territorial contexts. Related to this problem is the limited portion of the 
space that is framed by the images. In fact, this method only allows to detect the landscape 
elements that are framed in the images. The application of other methods that define landscape 
components extracting them from GIS database could be integrated in the procedure. The best 
foreseen solution could aim to define the most relevant components of a landscape extracting 
and defining them both from GIS databases (made of geometric entities derived from visual 
knowledge) and from Image Segmentation procedures, that feed a similar database, but using 
visual recognition of the same entities. In the end this double checked definition of entities should 
assure the correct definition of each item. Further developments could investigate the possibility 
to define a set of relevant objects referred to geometric entities (points, lines, surfaces) to be 
considered as common database records, so to be recognized and confronted both in the image 
segmentation based process and in its complementar GIS based method of analysis. 
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Abstract 


During this past year the Laboratory for eXtended Realities (DIDA-LXR) from the DIDALABS sys- 
tem at the Department of Architecture at the University of Florence, has experienced a various 
number of activities. Most of them linked together digital modelling of no longer existing architectures 
and still in place Built Heritage. Others were aimed to develop an “Augmented Virtual Reality” using 
specific environments/locations (for example a boat) to enhance the sensations of the user during the 
experience. Some others were based on direct VR shooting, using advanced panoramic cameras and 
creating a point of view compliant with the specific impressions that the place should transmit. 

In the contribution proposed here the AR, VR and XR experiences from this personal research will be 
presented sharing the specific subjects, the evaluation of usable technologies, the strategy for shoot- 
ing, survey, processing and post-processing, the dissemination of ideas and the lesson learnt. 
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Introduction 


The new possibilities created by the Laboratory for eXtended Reality were immediately fol- 
lowed up with case studies applied to cultural heritage. We consider the possibility given by 
virtual environments and augmented reality a great potential for reading and study, the way 
to increase and facilitate the knowledge of the architectural heritage. 

This technological development has led us to the need to deepen the existing possibilities of 
recording or surveying the existing. The growth of this knowledge has allowed us to think 
about topics that have a key theme and to experiment with new possibilities of fruition and 
use of the collected data. The whole set of experiences was oriented to Mediterranean 
subjects connecting the eastern and western parts of the “Mare Nostrum”. In this way, the 
Medusas Heads passing from Constantinople to Istanbul, the Tetrarchs’ Statue in Venice, 
the Venetian Lagoon and its story, becomes the first series of fragments of a VR and AR 
plot around the Mediterranean Sea. The approach is to create a versatile set of multimedia 
elements, oriented to work in a “classic” environment like the traditional display as well as 
in modern solutions like the immersive visors. At the same time, while experimenting with 
different solutions and subjects, specific attention was given to the users’ experience, gather- 
ing information and suggestions from the people using the virtual/augmented environments. 
This was quite useful in addressing and enhancing the further developments of models and 
proposals. The guidelines followed in all the testings and experiments were aimed at the 
production of very persuasive environments enriched by fascinating and valuable contents. 


From Site to Virtualization 


The paper presents experiences of eXtended Reality that aim to increase the user's senso- 
ry perception. This evolution is not sought by artificial technological inputs but by the real 
environment. Analyzing and structuring the real physical location where to place the device 
for the use of the digital environment we aim to add real inputs to complete the virtual ex- 
perience. From this, the cases presented are strongly different and distinguishable but with 
a common theme. The research wants to focus on the need to link the place of the fruition 
of the virtual environment with the digital product. This is to have a more complete prod- 
uct. However, do not want to take away the importance of the greatest potential of Virtual 
Reality, that of relating to the object of study in an environment completely parallel and 
independent from physical reality, releasing the digital object from the real world. The rea- 
soning wants to underline how eXtended Reality is not only the trivial union of AR and VR 
but it finds complete fulfilment in the knowledge and in the dialogue with the environment 
in which the experience takes place. Not only structuring and engaging with the location but 
also welcoming (and possibly shielding) the stimuli and features that are part of it. 


Case Study One: the Theater of the World by Aldo Rossi 


The first step in our journey to rediscover the Mediterranean Sea is Aldo Rossi's Theatre of 
the World [Brusatin & Prandi, 1982]. This iconic temporary architecture was built between 
the late seventies and early eighties and it has represented the symbol of what is known as 
temporary installations. Created specifically for the Venice Biennale of Architecture in 1979 
[Garcia 2006] still recalls more than 30 years later, the typical charm of something that was 
once there and now no longer exists, if not virtually. The purpose in this research is to keep 
this feeling of curiosity alive and untouched and then fulfil it, thanks to the development of a 
virtual environment, in which it has been relocated the Theater. 

The process behind the virtualization began with the documentation about the architecture 
and it was necessary to collect information about the structure and the form of the building 
itself, using architectural drawings and various photographic sources. Thereafter it was possi- 
ble to carry out the digital modelling of the object, performed with Maxon Cinema 4D and 
then exported in Unreal Engine for the virtualization of the environment. The experience 
led to the digital rebirth of this disappeared architecture (fig. 1). 
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Fig. |. The theater of the 
World reconstructed 

in its virtual space using 
Epic Games Unreal 
software, view of the 
virtual tour in the 
Basilica Cistern; view 
from a Panoramic VR 
shot taken in Poveglia; 
detail of the Tetrarchs 
statue in Venice from the 
sketchfab.com platform. 


The virtual tour was also designed as a place of education for students, including informa- 
tion points making the experience active. Following the example of the Biennale, it was 
decided to set up a temporary installation on a boat to create the dialogue with the real 
environment using the external inputs given by the location itself, and in this case, the 
movement of the boat and the sound of the water. 

Part of our research is also focused on another type of reality that allows us to create 
an overlay of the reality we find ourselves within. The model was reshaped and re-tex- 
tured and the process was carried out specifically to obtain a digital model suitable for 
augmented reality and dedicated software developing a beta version of an application for 
IOS devices using ARKit. The aim was to showcase the effectiveness of Augmented Reality 
technology as a powerful tool that can be successfully applied to the research process and 
the communication of the Cultural Heritage. 


fi En On in in im T 


‘Sunset in Poveglia, Venice Lagoon, VR 8K 


Case Study Two: The Medusa Protomes Inside the Basilica Cistern 


It is safe to say that to know better Venice it is essential to know Constantinople. The 
two cities were strongly linked by commerce, art exchange and share some robust envi- 
ronmental aspects. The Basilica Cistern, in Turkish Yerebatan Sarnici, is one of the largest 
ancient sites below the ground of Istanbul, built by the Emperors to satisfy residents’ 
water needs. Inside the building, under two of the 336 columns, we can find the Medusa 
Protomes, elements of interest not only for their history but also for the mystery about 
their origin [Verdiani et al. 2019]. The reconstruction of the Basilica Cistern was based 
on existing surveys, creating a digital model, optimally proportioned and that responds 
to the real [Ricci et al. 2018]. The Unreal Engine software was used to rebuild the Cis- 
tern in a virtual environment and to offer a format that can be reused in any museum 
context. Inside the tour the users have access to a series of multimedia containing in- 
formation about the site, about the myth that lies behind the medusa protomes and the 
recontextualization hypothesis proposed. This is the exact demonstration of the quality 
of this kind of result, which is an active type of communication that offers the simulation 
of areal environment and an unforgettable experience enhanced in this case by the uti- 
lization of some recorded sounds captured inside the Cistern during the survey. A short 
video about the virtual tour in the Basilic Cistern in Istanbul can be seen in Youtube at 
the following link: https://youtu.be/LsrxRaDrAo0. 
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Case Study Three: Venice the Tetrarchs Statue 


The Tetrarchs statue in St. Marco Square, Venice, is a clear sample about reuse of spolia 
as a demonstration of continuity and tells a yet partially mysterious story of artworks 
moving around the Mediterranean during strong and dramatic historical events [Dorigo 
2014]. Such an element, stable and solid across the centuries, but at the same time 
capable of a long and adventurous journey in its past [Missagia 2015], rich in referenc- 
es and symbols, apparently capable of hiding some intriguing stories and worth to be 
admired in tts details [Rees 1993], was found ideal for mixing two different modern 
digital approaches. A VR 360 short movie, recorded in timelapse mode was taken just 
in front of the statue, in a quite crowded hour, in the moment when the light of the 
sun is moving and the shadows are going to cover the corner. This short video clip was 
built to enhance the perception of the place with the special relationship between the 
static pose of the Tetrarchs and the fluid movement of tourists and workers all around 
the square, with the option of having a fully dynamic 360° point of view. The shooting 
was operated using an Insta360 Pro Il camera, capable of taking 8K resolution movies 
and stitching them in real-time. 
The video can be seen on youtube in the DIDA-LXR channel at: 

https:/lyoutu.be/xqvE | Huigjc. 

To satisfy the need of details and allow a complete exploration of the sculpture, pho- 
togrammetry was operated using a Nikon D800e. The full set of shots was then pro- 
cessed using Reality Capture software and producing a fully textured model of about 
500 million triangles, later simplified to five million and loaded with high resolution 
texturing on the Sketchfab.com platform and visible at https://skfb.ly/6UZ8P. 

These two digital products connected each other using simple links, allow a double and 
quite different reading and seeing this special corner in Venice, offering a better under- 
standing of the context and specific details in a way otherwise impossible in place, a 
valid alternative to the direct visit, but also stimulation and invitation to go in place and 
complete the knowledge about this migrating stone. 


Case Study Four: Poveglia, the Abandoned Island 


On the occasion of a recent trip in Venice, it was discovered a particular tiny island 
located inside the Venetian Lagoon named Poveglia offering the opportunity to test 
and discover the potential of a new tool, the Insta360 Pro 2 camera on a very fasci- 
nating environment where historic elements and urban legends are mixed [Cavallo & 
Visentin 2020]. After having chosen the right spots from which shot and film, the data 
was processed and it has been created a 360 tour that gives everyone the chance to 
visit the island. lt has proved to be a very useful tool and methodology for the dissem- 
ination of information about the existing heritage. 

A series of Panoramic VR videos can be seen in Youtube in the DIDA-LXR Channel, 
at: https://youtube.com/blaylist?list=PLBSGHBSIDCa-u 7-e | CQrAvQ | nEmpKZLKk 
Speaking of useful tools for surveying and photogrammetry, which in recent years have 
made significant advances in technology and functionality, we can not help but think of 
drones (UAV — Unmanned aerial vehicle), and the support they give in a survey thanks 
to their potential and the ability to access a point of view previously unthinkable. 

It is possible to recreate a 360 panorama, later navigable, using the flight application of 
the aircraft. The method is quite simple, taking advantage of the panoramic mode — 
360, the drone, autonomously, takes a series of photos in about 40 seconds in order to 
develop a navigable overview through the drone's piloting APP: FreeFlight 6. A sample 
of the result obtained from this procedure can be seen in the DIDA-LXR Youtube 
channel at: https://youtu.be/cjJz37]YDAU 

For the flights the drone used was a Parrot Anafi. The drone weight was reduced to 
300 grams in order to perform non-critical operations also in an urban context, as 
required by the regulations in force at the time of the flight. 
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Conclusion 


The present paper is composed by a sequence of studies with highly diversified contents. As 
stated in the initial part of the paper, the relationship with the resulting digital products has been 
accurately analyzed, trying to increase the virtual experience, creating a direct network between 
the fascination of the stories, the impression coming from the real places and the option offered 
by the digital solutions. 
The desire of experimenting and the will of building specific and effective results is at the base of a 
research aimed to find efficient solution for presenting parts of the Built and Cultural Heritage in 
the way they deserve exploiting their best characteristics, creating a specific digital version that is 
an enhancement or an alternative to the real but that doesn't ask the user to settle for quality and 
experience, it simply provide a digital approach offering the digital version of that cultural values. 
The case of the Theater of the World is totally concepted as a dialogue between the location and 
the virtual reality, increasing the senses’ suggestion. In this sense the statue of the Tetrarchs has 
opposite features, creating interest in something existing as a rich evidence of interactions in the 
Mediterranean area, the detailed photogrammetry links the real existing with virtual perceptions 
of knowledge and (maybe) with a desire of completion and investigation about the mysterious 
and impossible to be told story of its moving from Costantinople to Venice. The other two 
themes provide the possibility of new points of analysis, which move away from the theme in the 
studio but still keep close contact with the stimulus coming from the real, such as the introduction 
of recorded environmental sounds, in the case of the Protomes of Medusa. 

This set of experiences have a value that goes far beyond describing the themes and illustrating 
technologies and methods. They are traces of the experiences and above all provide “a way” 
about sharing points, suggestions, fascinations, they try to create a specific link between the inner 
value of stories and places in the continuous tentative of enriching the digital contents with valu- 
able learning occasions. Not only in the will of “capturing” the users, but in the true belief that 
“Digital Heritage” is an occasion allowing an easier reading and an option for spreading the knowl- 
edge of Cultural Heritage, but it need a correct comprehension of these values which is the first 
real step in creating a constructive system based on a sort of circular flow of attraction —> explo- 
ration —> knowledge —> digitalization —> dissemination —> learning —> spreading —> attraction. 
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Abstract 


This paper investigates the ephemeral architecture of the eighteenth century in Naples through the 
literary and iconographic sources found in the volume Idee per le pubbliche feste nel ritorno in Napoli 
de'nostri Augusti Sovrani dalla Germania [...] by Gaetano Barba (1971).The goal is an unprecedented 
representation, modeling and fruition project, accompanied by a graphic visualization of the places to 
better understand the architectural and urban space in the absence of allusive images of three-di- 
mensionality. Subsequently, the visual construction of the contexts on an urban and architectural scale 
made it possible to disseminate the compositional elements of the celebrations and the internal per- 
ceptions to a non-specialist public. Finally, an Augmented-Reality application prototype. 


Keywords 
ephemeral architecture, celebrations, representation, 3D modeling, augmented reality. 
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Through the synergy between the disciplines of Drawing and Technical Physics, this contribution 
analyzes the methods of graphic representation and the digital modeling of the elghteenth—cen- 
tury celebration designed (and not realized) in 1791 by the architect Gaetano Barba (1 730- 
1806) in Naples and described in the volume entitled: Idee per le pubbliche Feste nel ritorno 
in Napoli de’ Nostri Augusti Sovrani dalla Germania dell'architetto Gaetano Barba, Accademico di 
merito di S. Luca, edito a Napoli nel 1791 da Gaetano Raimondi. 

The results achieved with the graphic analysis and the build of new images are intended to 
represent, thanks to the aid of digital representation methods, both a cultural contextualization 
of the methods of representation used, and the return of a three-dimensional model to be 
understood as a phase of a subsequent study aimed at verifying the new technologies of virtual 
representation at the service of cultural tourism. The goal is to make the design ‘augmented’ so 
that the visitor can relive the context of the | 8th century Neapolitan celebrations. Therefore, it 
is proposed as an application case, the virtual reconstruction of the Porta di ingresso sulla strada 
di Toledo and its implementation in an Augmented Reality application. The application conceived 
for this purpose was created with the support of SENS Lab, a human—centered, multi-phys- 
ical and multi-purpose university laboratory for the creation, development, prototyping and 
interaction of man with products and physical and virtual systems of the Department of Archi- 


+ 


tecture, Industrial Design of the University of Campania Luigi Vanvitelli [1]. 


The Drawing of the Ephemeral: the Regulating Principle of the 1971 Celebration in Naples 
by Gaetano Barba 


The ephemeral concept has been (and continues to be) for the city of Naples a constant cultural 
expression, which is poured out not only in the everyday life of the people but also, and above 
all, in its architecture. The design of ephemeral architectures (such as fairs, cuccagne, pyrotechnic 
machines) saw widespread use in Naples during the eighteenth century with the creation of 
installations with a considerable design value [Cirillo 2017, pp. 101-118]. 

The theme of the ‘urban’ celebration reached one of its most significant moments in the eigh- 
teenth century with the staging of ephemeral settings that manifested the greatest affirmation of 
the culture and Art of that time. Their realization relied on a great variety of technical (architects 
and engineers), figurative and entertainment skills (musicians, painters, set designers). This synergy 
highlighted the taste for spectacularity grandeur sumptuousness and the decoration of the appa- 
ratuses to arouse the effect of wonder [Mancini 1997]. 

Within this context, in August 1790 in Naples, in the place called Largo di Palazzo (the open 
space in front of the Royal Palace) and along Via Toledo, a cycle of festivities was celebrated in 
honor of the return of the kings of Naples, Ferdinando and Maria Carolina, who were in Vien- 
na for the wedding of their daughters Maria Teresa and Maria Luisa. To celebrate this return, a 
competition was held between the best architects of that time. The celebrations’ principles were 
banned and published in the volume entitled Nel felicissimo ritorno degli Augusti Sovrani Ferdinando 
IV e Maria Carolina d'Austria. Feste Pubbliche della Fedelissima Città di Napoli (1791) at Giuseppe 
Maria Porcelli Librajo, e Stampatore della Reale Accademia Militare. The aforementioned volume 
reports as winner the set designer of the San Carlo theater Domenico Chelli, but there is no 
iconographic documentation about his project. However; Chelli, also with small formal changes, 
was inspired by the project outlined by the architect Gaetano Barba published in the same year 
in the volume Idee per le pubbliche Feste nel ritorno in Napoli [...] [Mancini 1980, p. 331]. 

The volume Idee per le pubbliche Feste nel ritorno in Napoli [...] by Gaetano Barba accompanies 
the reader to understanding the events not only through the reading of the short text but 
above all through the attached images. The images contained within the volume consists in ten 
engravings of the architectural elements, which set up the celebration. These representations 
appear ordered according to a spatial sequence which, starting from the Porta di ingresso sulla 
strada di Toledo and through the Sedili located along Via Toledo, leads to the rear scene of the 
Tempio dedicato alla Fortuna reduce, set up in Largo di Palazzo (opening figure). All the engravings 
show the architectural organisms represented in plan and/or elevation with full adherence to 
the neoclassical style [Jacazzi 1995]. The architect offers a general description of all the ele- 
ments of the celebration and outlines both their appearance and their functions and destinations. 
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Fig. |. Above, Entrance 
Door, Seats and Temple 
of Fortuna Reduce: plan— 
altrimetric drawings and 
3D modeling; below, 
visualization of the 
Temple of Fortuna 
Reduce in Largo di 
Palazzo (drawings and 
graphic elaborations by 
Pasquale Valentino and 
Vincenzo Cirillo). 


The installation of the entrance door was placed where the Ancient Royal Gate, also known 
as the Spirito Santo (current south side of Piazza Dante). Later, the celebrations would continue 
along Via Toledo, which was to be adorned and decorated with six Sedili (seats), called Ornati 
Architettonici di rilievo, which would have had the task of embellishing and making the facades 
of the street more pleasant. 

The six seats suggest that they would have been arranged with the long side parallel to the road 
axis. In this sense, it is plausible to think that these structures would have leaned on the elevations 
of the buildings on Via Toledo, both for the lack of a second elevation in the back and for the anal- 
ogy with other festivals such as, for example, that of Santa Rosalia in Palermo where ephemeral 
structures were placed on the facades of existing buildings [Isgrò 2019; Di Fede 2005-2006, pp. 
49-75]. In Largo di Palazzo, a majestic Temple dedicated to Fortuna Reduce would have concluded 
the festive itinerary. The temple with colonnades, a large central staircase, two obelisks, statues and 
two arms of pedestals surmounted by statues of sirens, would have stretched towards the statue 
of the Giant and the Church of the Santo Spirito (demolished), thus redesigning the planimetric 
layout of the square with a regular shape. The drawings by Gaetano Barba allowed a correct and 
meticulous analysis and representation of all the elements designed, thus succeeding in grasping 
the dimensional and spatial character of each structure. In this sense, starting from the literal 
description, from the project drawings and from a conversion into meters of the dimensions 
indicated in ‘Neapolitan palms’, through graphic analysis, modeling and subsequent visualization 
(fig. |, above) an unpublished image is returned. of the event according to the criteria and ideas of 
Gaetano Barba respecting the urban context of the Neapolitan eighteenth century (fig. |, below). 
Given this, the digital reconstruction of the entrance door and the virtual location in the current 
Piazza Dante through the AR application constitute the case study for visualizing both the party 
imagined by the designer and the actualization effect. 


Wn 


161 


Fig. 2. Above, optimization 
of the 3D Model and 
creation of Assets; below, 
Implementation of the 
plan recognition system 
for AR applications (by 
Aniello Pascale). 


The Prototype of the Application in AR 


The development of digital content that can be used through mass-market devices (e.g. mo- 
bile phones, tablets) and head-mounted displays in the last decade has found a fertile field 
of application in the world of Cultural Heritage [Voinea 2018]. More and more varied and 
interactive digital contents, accompanying the offer to tourists, are having increasingly interesting 
implications: (1) the possibility of visiting sites that no longer exist or are inaccessible represents 
an opportunity for museums to expand the offer of contents and cultural sites, with the ability 
to attract and intrigue even a younger user target; (2) the digital contents are conveyed to the 
end-user directly, providing them with a more or less immersing experience according to the 
types of devices in use. (3) The digital contents are made available to the user through their 
smartphone, with the possibility to contextualize it in its original place. Taking advantage of the 
opportunities deriving from Digital Cultural Heritage applications, an App prototype has been 
created. Through the Augmented Reality (AR), the App allows users to walk through the streets 
of a city, its monuments, artefacts, lost buildings or in any case, not accessible. The first application 
concerned the artefact of the entrance door sited in Via Toledo in AR. The workflow sees a first 
phase dedicated to creating the contents necessary for developing the App: Three-dimensional 
model, Texture, Sounds. The three-dimensional model was made starting from the drawings 
recovered in the bibliography, as described above. In this phase, more attention was focused on 
defining the details of the 3D model: frames, moldings, capitals, rather than on the materials. It 
was important to create a reasonably light model in terms of ‘vertex counting’ which, however, 
did not compromise the elements’ definition. The complete model in all parts was then export- 
ed to the 3DsMax software for polygon optimization and asset preparation. In particular, the 
elements that make up the product have been appropriately divided, and IDs have been assigned 
for the diversification of materials. The textures were downloaded from Megascans libraries. As 
for creating sounds, a sound effects composition operation was carried out, starting from pro- 
fessional libraries’ sounds. Anthropogenic sounds such as pedestrian footsteps, people's voices, 
and music have been downloaded to recreate the partys typical sound setting. The sounds have 
been converted all mono and imported into the development environment. Here they have 
been inserted as points sound-source so that you can make a spatialization of the sound. The 
App was created within the ‘Unreal Engine 4’ development environment. The ARCore framework, 
necessary for the recognition functions of surface planes in AR mode, was implemented here. 
A basic interface has been inserted that allows the user to enable the camera and frame the 
horizontal plane where the 3D model will be displayed. The actor was then created to be placed 
on the surface, with the artefact's 3D model inside. Once the floors are recognized, a tap on the 
screen allows you to position the object; subsequent finger gestures will enable you to move and 
rotate it (fig. 2). A first test was carried out directly on site. In piazza Dante, near the beginning of 
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Fig, 3. From left, Micco 


Spadaro (XVII secolo), 


Punizione dei ladri al 
tempo di Masaniello; 
Insertion test of the 
artefact in the original 
site. 


via Toledo, the artefact was positioned. After recognizing the support surface's height, the opera- 
tor could place the artefact easily and explore it from the most congenial point of view. During 
the experience, the sounds, located almost under the arch, helped to realize what Porta Toledo's 
installation must have been, in the place where it was designed (fig. 3). 


Conclusions 


On the basis of the sources collected and through the discipline of drawing, this paper analyzes 
the project of the ‘Pubbliche feste [...]' by Gaetano Barba (1791). Accompanied by unpublished 
graphic elaborations, the paper describes the architectural elements that make up the cele- 
bration through an ‘augmented’ design. In this sense, the virtual reconstruction of the entrance 
door on Via Toledo and its implementation in an Augmented Reality application was proposed 
as an application case. The choice to contextualize AR to the current urban situation instead 
of set it in the late eighteenth century results from the fact that the research group is analyzing 
the design and reconstruction of other eighteenth-century Neapolitan celebrations. Therefore, 
the reconstruction of the eighteenth—century city will be formulated as future works where 
will be evaluated the different urban places, sometimes, modified in relation to the ephemeral 
architectures construction. 


Notes 


[I] The research presented is the result of the joint work of the five authors. The paragraphs Introduction and Conclusions 
Is written by Ornella Zerlenga and Luigi Maffei; the paragraph The drawing of the ephemeral the regulating principle of the 
1971 celebration in Naples by Gaetano Barba by Vincenzo Cirillo and the paragraph The prototype of the application in AR by 
Massimiliano Masullo and Aniello Pascale. 
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Abstract 


The Charterhouse of Pisa in Calci, one of the most imp 
two important museums: the Natural History Museum 
Museum of the Monumental Charterhouse of Calci. Whi 


ortant monasteries in Tuscany now houses 
of the University of Pisa and the National 
e the Natural History Museum has recently 


enriched its collection by offering structured and differentiated visits based on user type, the offerings 
of the Museum of the Monumental Charterhouse are not sufficiently adequate to meet the great his- 


torical value of the complex. This contribution therefore 


presents the first results of a project aimed 


at enhancing visits to the National Museum of the Charterhouse using immersive technologies. The 
project envisages the definition of a new visit path, modifying the current path and integrating it with 


immersive experiences of video mapping, VR/AR, sound 
supports, and multisensory activities. 
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Fig, |. Frame of the 
introductive video 
projected in the Chapel 
of San Sebastiano 
(elaborations by A. 
Fedeli). 


Introduction 


Recent studies demonstrate how immersive technologies based on augmented, real, and 
mixed reality are currently and widely used in the fruition of cultural heritage (Bekele et al. 
2018). In the field of fruition of monuments and museal spaces, these technologies provide 
solutions enabling patrons to view 3D digital models of cultural artifacts and to interact with 
them in a variety of ways, enhancing their involvement (Trunfio et al. 202 1). Most of these ap- 
plications enjoy continuous evolution in the fields of 3D digital survey, modeling, and graphics. 
AR applications, virtual reconstructions, video mapping, etc. are, in fact, widely used to enhance 
visiting experiences, as well as serving as tools for promotion and enhancement of cultural 
heritage. In Italy, there are several noteworthy cases, such as those of the Virtual Archaeological 
Museum of Herculaneum, the Egyptian Museum of Turin, and the National Archaeological Mu- 
seum of Naples, the AR/VR applications at the Baths of Caracalla and the Ara Pacis in Rome, 
and the video mappings in the Imperial Forums in Rome, to name but a few. 

This contribution intends to present the first results of a project aimed at enhancing the 
tour of the National Museum of the Monumental Charterhouse of Calci using immersive 
technologies. The project envisaged the definition of a new visit path, modifying the current 
one and integrating it with immersive experiences of video mapping, VR/AR, sound immer 
sion, information totems, audio-visual supports, and multisensory activities. 


the porch and the aqueduct in the Cloister of the Fathers. 


The Case Study: the Museum of the Monumental Charterhouse of Calci 


Founded in 1360 on the slopes of Monte Pisano, the Charterhouse of Pisa in Calci rep- 
resents one of the most significant monasteries in Tuscany. The charterhouse is the result 
of several modifications and extensions documented from its founding to the end of the 
18th century and reflects the strict rules of the Carthusian Order [Piombanti 1884; Manghi 
191 1, Giusti & Lazzarini 1993]. It represents an ideal semi-urban ‘village’ in which hermit and 
cenobitic life are harmoniously blended. The correria, the coenobium, and the desertum are 
distributed following the idea of a gradual separation of the fathers from the common life. 
The large Courtyard of Honor separates the entrance and the correria from the coenobium 
and the church; this is placed in a central position emphasizing the compositional axis of 
the entire complex. On the southern side of the church, there are spaces dedicated to the 
life of the religious community (minor cloisters, refectory, colloquium, and chapter room), 
the noble guesthouse, and the grand ducal apartments. The northern side boasts several 
buildings originally used for agricultural activities. The Great Cloister with the cells of Fathers 
completes the system on the eastern side. 

In 1962 the religious community left the Charterhouse of Calci. Since 1972, the complex has 
hosted the National Museum of the Monumental Charterhouse of Calci, under the control of 


168 


Fig. 2. Orthophotos of the 
frescoes in the Chapel of 

St. Anthony (elaborations 

by A. Fedeli). 


the Ministry of Culture and, since 1978,the Natural History Museum of the University of Pisa. 
The National Museum of the Monumental Charterhouse offers visits to the monumental 
spaces of the monastery: the church, the refectory the chapter room, the noble guesthouse, 
the grand ducal apartments, the pharmacy, and one of the fathers’ cells. The Natural History 
Museum occupies most of the service buildings on the northern side of the complex. 

The Natural History Museum has recently enriched its collection and provides customized 
visits based on user type; meanwhile, the current offerings of the Museum of the Monumental 
Charterhouse aren't sufficiently adequate to the great historical value of the complex, pre- 
senting critical issues: some of the most important spaces are not visitable, visits are organized 
in staggered guided tours, and contents are oversimplified. 

In 2018, the University of Pisa funded an interdisciplinary research project aimed at the con- 
servation and enhancement of the Charterhouse [|]. More than 15 research units participat- 
ed in the project, developing research in several fields, from historical analysis to specialized 
studies, such as those relating to fire prevention and accessibility. 

As part of the project, in addition to a campaign of digital surveys and an in-depth historical 
study, the work we present here was developed, aimed at enhancing visits to the National Mu- 
seum of the Charterhouse through immersive experiences of augmented and virtual reality. 


The Project for a New Guided Tour of the Museum of the Charterhouse of Calci 


The project was structured in two phases. The first concerned preliminary analysis for the 
development of the project. In addition to a study of the state of the art in the field of VR/ 
AR use in museum environments, the tour path currently active in the museum was analyzed 
to highlight its problems and potential. The results of the in-depth historical research on 
the Charterhouse, developed under the coordination of Ewa Karwacka within the project 
funded by the University of Pisa, provided the definition of objectives and contents of the 
new visit tour Specifically, it was decided to focus the tour on the monastery's evolutionary 
phases, on its most valuable decorative and architectural elements, and on the figures who, 
over time, had a prominent role in the history of the Charterhouse. 

The second phase concerned the development of the project. The new guided tour is based 
on the current one but modifies its route, expanding the number of spaces that can be visited 


169 


Fig, 3. Comparison 
between real space — on 
the top — and illusory one 
— on the bottom — in the 
Chapel of Sant'Antonio. 
Plan and longitudinal 
section (elaborations by 
A. Fedeli). 


and increasing the immersive involvement of visitors, who are left free to move along the path 
using an audio-video guide. The guide can be downloaded on tablets or smartphones and 
provides information on the spaces and their locations. Each space features an informative 
totem, and sound atmospheres are envisaged to amplify the sense of interaction. 

Access is scheduled at set times for groups of people who, once their entrance tickets have 
been paid, can wait for their turn in the nearby San Sebastiano Chapel, appropriately rear- 
ranged for the projection of an introductory video on the history of the Charterhouse and 
the life ofthe monastic community. 

All the spaces are classified into three categories based on the information that will be provid- 
ed: art and architecture, monastic life, and mixed information. In some spaces, experiences of 
virtual and augmented reality, video mapping, or multisensorial experiences are provided. All 
the applications are accessible for deaf people as well, through a guide in Italian sign language. 
Once the general visit program was defined, some of the augmented reality applications were 
developed: the introductory video projected in the Chapel of San Sebastiano, the virtual 
reality experience in the Chapel of Sant'Antonio, and that in the Cloister of the Chapter [2]. 
For the development of the applications, realistic 3D models were created on the basis of 
the results of the surveys carried out with LIDAR and 3D photogrammetric methodology 
by the ASTRO Laboratory within the activities of the project funded by the University of 
Pisa. Models were developed with the open-source software Blender. The informative texts, 
based on the original results ofthe historical research, were processed with Audacity a soft- 
ware for editing and audio recording, The final processing was developed in graphic anima- 
tion software including Adobe After Effects for the introductory video in the Chapel of San 
Sebastiano, and Unreal Engine for the immersive experiences in the Chapel of Sant Antonio 
and in the Cloister of the Chapter. 
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Immersive Experiences at the Museum of the Charterhouse of Calci 


The first digital application developed was the introductory video projected in the Chapel 
of San Sebastiano for visitors waiting for entry. The historical research developed in the 
project of the University of Pisa identified, in chronological order the most significant events 
that affected the Charterhouse from the day of its founding to today. The video presents a 
graphic animation of 3D models that reproduce an external view of the various construc- 
tion phases of the complex. Information pop-ups superimposed on the video help to point 
out the buildings as they are being described in Italian by the narrator voice. Subtitles scroll 
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for the translation of the text into various languages; a small panel in the lower corner of 
the screen is dedicated to the video guide in Italian Sign Language for deaf visitors (fig. |). 
The second application concerns the experience of virtual reality in the chapel of Sant’An- 
tonio, characterized by the quadraturist frescoes by Pasquale Cioffo, a Neapolitan painter 
very active in Pisa in the second half of the | 8th century. The experience focuses on quadra- 
turist painting techniques, explaining the geometric principles underlying the representation 
and providing a 3D reconstruction of the illusory space depicted (fig. in cover page), inside 
which the visitor is immersed thanks to the use of a VR headset. Wearing the headset, vis- 
itors find themselves inside a high-resolution 3D reconstruction of the real environment, 
created thanks to 3D photogrammetry techniques (fig. 2); the narrator voice explains what 
is being observed. Subsequently, the real environment changes and the reconstruction of 
the illusory space imagined by Cioffo appears (fig. 3). The story then focuses on the optical 
phenomenon of anamorphosis, found on one of the sides of the chapel. 

The last application is the virtual reality of the Cloister of the Chapter, also to be experi- 
enced with aVR headset. In this case, the viewer is immersed in virtual spaces that depict 
the layout of the cloister in the most important historical phases; a narrator voice accom- 
panies the visitor during the experience. The development of the application required a 
preliminary reconstruction of the 3D models that describe the various evolutionary phases. 


Conclusions 


The project aimed to investigate the potential in the application of new AR/VR technologies 
to the specific case of the Charterhouse of Pisa in Calci. This work was also an opportunity 
to study a communication strategy for the results of the research project funded by the 
University of Pisa with particular reference to that of the historical/critical study. The first 
results obtained in this phase are functional to the search for funding for the realization of 
the project as a whole and of the specific applications. 


Notes 


[I] The research is part of the biannual research project “Studi conoscitivi e ricerche per la conservazione e la valorizzazione 
del Complesso della Certosa di Calci e dei suoi Poli Museali" financed by the University of Pisa in 2018 and 2019, coordinated 
by M.G. Bevilacqua. The authors wish to thank Ewa Karwacka, Elisabetta Pozzobon and Stefano Aiello, director of the National 
Museum of the Charterhouse of Calci, for their support and collaboration. 


[2] Video demonstrations of the applications are available at the following links: https://www.youtube.com/ 
watch?v=haLQXXsLZ_k (Chapel of San Sebastiano), https://www.youtube.com/watch?v=pPZuzoBbaRQ (Chapel of 
Sant'Antonio), https://www.youtube.com/watch?v=f4vtJOBrqMw (Cloister of the Chapter). 
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Abstract 


The paper shows the results of the PRIN CHROME Cultural Heritage Orienting Multimodal Experi- 
ences project, about the three charterhouses of Campania, with a specific focus on research activities 
related to the connections between representation, survey, Al and VR. The project has formalized 
a methodology of collection, analysis and modeling of multimodal data, useful for designing virtual 
agents in 3D environments, which can be applicable in museum environments. The achievement 
of the goal is pursued through: (i) an integrated range-based acquisition and morphometric data 
modeling process coherent with VR management, (ii) the use of semantic maps linked with thesauri 
published as LOD to solve both the theme of ambiguity and annotation uncertainty and the inter- 
pretability of information by an Al; (iii) the modeling of a virtual agent with the development of a 
mathematical model for computational control of gestures and prosody. 
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The CHROME Project: Methodology and Procedures 


The paper shows the specific results of the PRIN 2015 CHROME Cultural Heritage Re- 
sources Orienting Multimodal Experiences, developed around the case studies of the three 
charterhouses of Campania, with the focus on research activities related to the connections 
between representation, survey, artificial intelligence and virtual reality [1]. 

The project, that is strongly inter-disciplinary, has formalized procedures for collecting, anno- 
tating and analyzing multimodal data — such as written texts, oral presentations, 3D models 
— for a subsequent use by the Al. 
n particular the resources collected and annotated have served to design a virtual agent 
inserted in 3D virtual scenarios. This Virtual agent can be applicable in museum environments 
and joins the tour guides increasing the potential for intervention on the public visiting cultural 
sites. The virtual agent, in fact, simulates social signals through computational control of ges- 
tures and prosody according to a mathematical model based on the behavior of operators 
specialized in the communication of cultural contents. 
The base knowledge has therefore been structured in order to build a model that allows to 


+ 


compose a not default and potentially adaptable to the type of interlocutor oral presentation. 
The achievement of the goal was pursued through the semantic association of the whole 
corpus of information to the geometric entities of the spatial model, that are digital clone of 
the real good to which the enhancement is addressed. The annotation of 3D representations 
made it possible to link the presentation to the automatic selection of the auxiliary material 
and to query it with a natural language dialogue system, in which the information is spatialized. 
As disclosed here, attention is focused on those investigative activities related to the inter- 
connections gained between the disciplinary of representation and survey and the domain 
of computer science, related to each other and put at the service of the development of Al 
applications in augmented and virtual reality environments. 

Since this background, the specific research investigated the theoretical and methodological 
issues related to the geometric and semantic manipulation of digital representations of ar- 
chitectures or rather; on one hand, those of a terminological-significant nature and, on the 
other, the ones of geometric-formal matrix. The first ones involve the process of meaning 
assigning to spatial forms, the latter concern both the processes of “construction” of the dig- 
ital clone and the method by which recognizing on it the geometric boundary of semantic 
concepts. In addition, the considered segmentation approaches have been strictly aimed at 
storing content in an Al-questionable system, made able to disseminate information in digi- 
tal settings that can be experienced through AR/VR technologies. This last aspect involved a 
reflection upon the most appropriate ways of graphic simplification of the elements of the 
heritage in order to make their vision fluid in a system of virtual use without losing neither 


the realistic rendering nor the understanding of the contents. 


Representation for Semantic Structuring and Knowledge Formalization 


The first phase of the study involved the realization of the digital virtual scene, to be seman- 
tically annotated, for subsequent use by the Al. 
For the three case studies of the project, the charterhouse of San Martino in Naples, the San 
Lorenzo one in Padula and the San Giacomo one in Capri, important campaigns have been 
carried out. These have seen the integration of passive and active optical sensors in order to 
achieve accurate, precise and photorealistic three-dimensional models, returning both of the 
overall morphology of the different convents and the complexity of the decorative details 
of the interiors. 
Starting from the integrated range-based acquisition of morpho-metric data, point clouds 
were modeled with classic triangulation algorithms and subsequent texture projection. The 
models obtained from the multi scalar survey were then developed for rendering in intensive 
D application development environments, initially subjecting them to a process of selective 
decimation of the level of detail to make their vision fluid and then subjecting them to a pro- 
cess of texture baking to not lose their realistic output or the understanding of the contents. 


(99) 
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Fig. |. Semantic maps 
produced by different 
annotators for the 
concept of capital. 
Calculation of the degree 
of agreement with its 
final map. 


To link 3D models to the Al, annotations have been added or rather questionable metadata 


that encodes the knowledge of domain expert 


In relation to domain vocabulary, the art and archit 


architecture world formalized by the Getty Institut 


format designed for compatibility with triple RDF, 
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different data sources to allow Al to efficiently c 


from Wikidata. 


n order to associate semantic concepts with 
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the information related to the process of 
d into a purely visual image [Cera 2018]. 


The possibility of using a grey scale allows to refine the quality of information where the 


gradations have different percentages of 
consider semantic maps produced by multi 


relevance. So, this method makes it possible to 
ple domain experts obtaining a final map of their 


degree of agreement, calculating the average values for each UV coordinate (fig. |). 
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The innovative thing of coding and using semanti 
tation margin of error which is almost always ignored in the usual processes of semantic 


c maps lies in allowing to manage anno- 


segmentation of digital representations. The margin of error of domain experts is, on the 
contrary, a substantial element in the knowledge process where it provides knowledge and 
complex cognitive mechanisms. For example, using this procedure, the margin of error due 


to the annotator's background, is not only reco 


rded but also turned into a resource. 


Machine learning approaches, in fact, are based on statistical models that, in this case, should 


model the probability of each geometric element 


t belonging to a given category. Discretizing, in 


representation, is equivalent to removing information from the data on which the model is build, 
thus imposing a ‘filtered’ view to the algorithm, which has no way of modelling the existence 
of concepts which, for example, fade into each other or whose definition depends on multiple 
factors, such as the specialization of the domain expert in charge of the annotation, the aim of 
segmentation, the support on which the recognition 
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process is implemented, etc. [Cera 2019]. 


From Survey to Development of Artificial Intelligence 


The semantic maps gathered with AAT codes, make the information contained in the map 
cross-referenced with that contained in the other resources annotated such as in the texts, 
in the AAT itself and in other LODs. 

To make access to information fast and efficient for interactive applications that use real- 
time 3D material, knowledge has been depicted within a graph database [Webber 2012], 
which drastically reduces latency due to querying online resources, for example in RDF 
format. This allows to quickly cross-check information from different sources and compare 
it to adequately support the application. 

Within a set of reference texts, the same concepts, described in geometry by semantic 
maps have been identified and annotated. In this way, you can associate, with the text that 
describes a resource, the geometry to which it refers independently of the application, 
making the material highly reusable for different purposes. 

One of the possible applications achievable with the type of annotated material is the de- 
velopment of conversational virtual agents placed in an environment about which they have 
sufficient knowledge to interact with them.To study the behavior that these agents should 
take, a corpus [Origlia 2018] of 12 hours of audiovisual material was collected to document 
the behavior of art historians who illustrate the environments of the charterhouse di San 
Martino to small groups of visitors. 
A linguistic and psychological annotation system has been created to cross—check the various 
levels of communication through which an experienced human transmits cultural content. 
In the laboratory, motion capture data was collected to map human movements to 3D 
avatars. The logic of managing the gestures of the virtual agent has been defined as follows: 
at each frame, the system calculates the position to be assumed on the basis of a series of 
animations that are combined according to a series of parameters. As far as the gestures 
of the arms are concerned, there is a dedicated state machine which places the agent in 
a ‘neutral’ position. When an externally produced signal arrives, which corresponds to the 
start of an audio containing a synthetic voice, the agent switches to ‘talking’ mode. During 
speaking mode, an external system may require highlighting concepts with varying degrees 
of ‘strength’ or pointing in a certain direction. Since the location of the virtual agent is known, 
the only information you need to control its gestures is the location of the target. 
Using the centroid of the mass of points labeled with a certain concept, for example “al- 
tare maggiore” imagining that the virtual agent is placed in the church of San Martino, it is 
possible to calculate the angle between the virtual agent and the concept that you want to 
point, thus providing the animation control system with the missing information to produce 
coherent dectic gestures. 

The processing pipeline that allows an Al to interact is made of several modules. 

First of all, a specially trained neural network transcribes audio containing a user's voice. 
From this transcription, an ‘intent’ is extracted, that is the abstract intention of the speaker 
and any parameters that detail the request. Based on intent and parameters, a graph data- 
base query is produced to extract the content needed to fulfill a request. The sentence to 
be synthesized is then passed to a second neural network that synthesizes the audio and 
produces the accompanying information, such as the phonetic annotation of the audio file, 
to allow the control of lipsync, and the indication of the temporal position of the expressed 
concepts, to control the gestures of the deictics. Based on this information, an interaction 
management engine delivers the presentation in real time. 


Results 


The research developed as part of the CHROME project provided an opportunity to 
investigate the increasingly structured interconnections between the field of representation 
and survey and the themes of information technology. In particular, the paper analysed the role 
that the specification of architectural survey and the forms of drawing play in the development 
of Al applications tested in the dissemination of cultural contents related to some architectural 
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heritage of Campania. Validated on the case studies of the three charterhouses, the project 
developed a method of collection, analysis and dissemination of spatialized information 
resources around three—dimensional architectural models, used in digital environments whose 
presentation is entrusted to virtual agents modelled on human behavior: 

CHROMES system architecture is designed to be generalized in a framework called FAN- 
TASIA [Origlia 2019] for developing conversational virtual agents that can be applied in any 
museum environment and therefore replicable (fig.2). The architecture uses graph databas- 
es to link data from different sources such as LOD, three-dimensional models, or other. It 
enables the use of modern peripheral devices and third-party services for capturing and 
analyzing input signals and integrates probabilistic decision-making systems for controlling 
interaction in 3D environments. 


Notes 


[I] The Pl of the Italian PRIN project CHROME #B52F | 5000450001 is prof. F. Cutugno. The architecture unit was coordinated 
by profs. M. Campi and A. di Luggo. Arch. D. lovane worked on architectural data acquisition together with arch. V. Cera who 
developed the research on semantics. Dr. A. Origlia worked on the All. development. 
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Abstract 


Starting from a multi-scalar and multi-dimensional survey, most interdisciplinary researches, based 
on representation, are becoming a tool for dialogue between the new trends of Artificial Intelligence 
(Al) and the most compelling needs of our CH. The approach here proposed stems from the desire 
to understand how much of the skills useful in architecture analysing and modelling can be made 
available to the “machine”, with the goal to accelerate cognitive or management processes. Some 
HBIM models, as an existing digital heritage, were used to obtain the semantic intelligence. From this 
specialised intelligence comes a cyclical path which, through Al, transforms this knowledge into new 
forms of collective intelligence, at the service of the heritage. The paper presents a research that 
brings very promising results for the segmentation of point clouds and the facilitation of ScanToHBIM 
approaches, made possible by the large amount of data acquired on the Ducal Palace of Urbino. 


Keywords 
semantic segmentation, Al, intelligent models, HBIM, collective intelligence. 
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Introduction 


The Digital Cultural Heritage (DCH) stands today as a cornerstone in the processes of 
Built and Museum Heritage management and knowledge but also in its conservation. Start- 
ing from the current practices of multi-scalar and multi-dimensional survey, ranging from 


+ 


andscape and monumental heritage to artworks, the interdisciplinary research, based on 
representation, is becoming a tool for dialogue between the new trends of Artificial Intelli- 
gence (Al) and the most compelling needs of our heritage. 

n the field of CH, Al applications are dealing with both enabling new forms of digital data 
management, both generating digital assets from existing ones, with a surprising capacity of 
mimesis. However, a theoretical debate on the implications of Al in the context of Digital 
Humanities and Computational Modelling is still lacking. 

The approach here proposed stems from the desire to understand how much of the 


+ 


skills useful in architecture analysing and modelling, inherent to the drawing discipline, can 
be made available to the ‘machine’, with the goal to accelerate cognitive or management 
processes with regard to the built heritage. Some HBIM models, considered as an existing 
digital heritage, were used to obtain the semantic intelligence. From this specialised intelli- 
gence comes a cyclical path which, through Al, transforms this knowledge into new forms of 
collective intelligence, at the service of the heritage. 

The work is part of a broader strand of research in DCH by UNIVPM, now significantly 


developed in the strategic University project CIVITAS which addresses several challenges 
related to museums and the historical buildings. 


State of the Art 


The architectural heritage documentation is based on point clouds, with accurate discrete data- 
bases. However in order to become effective, they need an informed and structured representa- 
tion, mandatory based on semantic subdivision. For this reason, point clouds are often used, 
within BIM software, as a starting point for building parametric 3D, which incorporates semantic 
information and where architectural elements are identified and enriched with non-geometric 
information. This process, namely ScanToBIM, is costly and requires skilled operators. 

The SACHER project obtained interesting BIM-independent results in the use of segment- 
ed 3D data [Bertacchi et al. 2018, pp. 283-288], while the INCEPTION projects [ladanza 
et al. 2019, pp. 381-388] and the BIM3DSG platform [Rechichi et al. 2016, pp. 703-710] 
achieved to interoperate and manage knowledge outside BIM platforms. 

The need to automate, at least partially, the ScanToBIM process is largely agreed, certainly 
not in order to diminish the designers knowledge but to make the whole process more 
agile focusing on new research challenges. Mainly the backbone is to make models more 
intelligent and aware of their nature. 
An important step is the semantic segmentation, facilitating the identification of different 
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types of architectural elements in the point clouds. It implies classifying each point to a pa 


de 


ticular type of object (e.g. wall, roof, column, vault, etc.). While Machine and Deep Learnin 
techniques are spreading in every field, even at the basis of ‘popular’ applications with image 
abelling, point clouds make the task much more complex, especially when point clouds pe 
tain to historical architecture. Machine Learning is also giving satisfactory results for dataset 
with different accuracies and hierarchical procedures [Teruggi et al. 2020, p. 2598]. To date 
deep learning is particularly challenging for classical architecture, due to the complexity of 
shapes and the limited repeatability of elements, making it difficult to define common pat- 


terns within the same class of elements [Pierdicca et al. 2020]. 
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The CIVITAS Project and Multi-Scalar Acquisition in the Ducal Palace at Urbino 


The paper presents a research that brings very promising results for the segmentation of 
point clouds and the facilitation of Scan ToHBIM approaches, made possible by the large 
amount of data acquired on the Ducal Palace of Urbino and the collection of the National 
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Fig, |. Outputs of SM 
surveys: bas relief by 
Francesco di Giorgio 
Martini and Duke's 
Studiolo 


Gallery of the Marche region. The digitisation phase of the building and museum artefacts 
is a fundamental step of the CIVITAS project, as detailed in [Clini et al. 2020, pp. 194-228]. 
One of the challenges, in particular, is the optimisation of data management dealing with 
HBIM exploiting Linked Data, Semantic Web and Artificial Intelligence technologies, in order 
to perform new workflow starting from reality-based informed models. 

The acquisitions implemented for the Ducal Palace of Urbino were based on a multi-scale 
approach, as highlighted in [Nespeca 2018, pp. 1-14]: they tested various types of instru- 
mental acquisition and defined the most appropriate levels of detail for the various forms 
of representation and features of the building. All the interior rooms and courtyards, but 
also the exterior of the building, were digitised with TLS, merged on a general 3D model 
acquired with Mobile Mapping and assisted by a campaign of spherical panoramas. 

For many paintings and sculptures of the collection and for the Duke's Studiolo, dedicated 
photogrammetric acquisitions were deployed (fig. |), to complete the set of acquisitions. 
In this way, a comprehensive three-dimensional mapping was conducted, which forms the 
basis for any scientifically based action in the process of digital transformation of museums. 


Semantic Segmentation by Deep Learning Approach 


The point clouds, with the various gathered accuracies (fig. 2), constitute a high—quality 
morphometric model, but the nature and pertinence of the single points to the different 
components of the architecture is inferred by humans. Till recently the subdivision of point 
clouds was mostly based on algorithmic or manual approaches, now the possibilities granted 
by neural networks lead to exploit them to recognise points and assign them a semantics 
consistent with the rules of the architecture. 

The identification of classes, which also has theoretical implications of great interest, has 
proved to be fundamental in the methodology. Following an analysis of coherence with the 
existing and consolidated thesauri and considering the historical period, its morphologi- 
cal-formal language and the features of the acquired data, a first level scale of the semantic 
hierarchy was defined (fig. 3). 

The state of preservation of the building gives reason to expect very good correspond- 
ences with ‘ideal’ forms. The subsampling required to perform several tests and in several 
epochs, with an average computational capacity, also led to the choice of working on the 
most general level of the main architectural members. 

A bottleneck in Deep Learning approaches applied to CH is the absence of sufficiently large 
annotated data sets that can allow the training of the networks. Thus, the approach took 
advantage of many previous models developed with semantic structuring features: from the 
Palladio Library villa models developed in 2012 to the most recent HBIM models (Ferretti 
Palace, Santa Maria of Portonovo and Ducal Palace). In addition, parametric models present 
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Fig. 2. Point clouds of the 
Ducal Palace at Urbino. 


Fig. 3. Identification of 
classes in the taxonomy 
about the Honour 
Courtyard of Ducal 
Palace. 


on the web were used and appropriately selected, as well as the HBIM core of the Honour 
Courtyard of Laurana. This allowed the generation of a sufficiently large data set of synthet- 
ic clouds to train the neural network. This phase therefore places drawing knowledge at the 
centre and the intelligence present in the models is regenerated and acquires new life and 
unexpected opportunities for value. 

All the chosen models had the common characteristic of being organised according to a 
shapes grammar in which the constituent ontologies of the architectures had been analysed 
and studied. In the models, the classes were also consistent with those of the point clouds 
that were to be segmented, both in terms of formal qualities and hierarchical articulation. 
All models were stored in the various formats (.rvt, .3dm, .kmz) and archived in a file format 
that incorporated the taxonomy by naming the layers. This enabled the export in .obj for- 
mat and the subsequent creation of semantically structured synthetic point clouds. 

At this point, the workflow for the Deep Learning approach foresees firstly the training of 
the neural network, i.e. its training, and then the experimentation on a never observed da- 
taset. The DGCNN network was chosen, which is based on the EdgeConv operation, and 
also a refinement of it RIDGCNN, as better detailed in [Morbidoni et al. 2020a; 2020b]. 
Two experiments were performed on two different case studies: in the first one, we used 
the TLS point cloud of the courtyard of the Ducal Palace of Urbino to evaluate the trained 
models, trying to identify 8 different classes of architectural elements. For consistency, the 
synthetic point cloud derived from the HBIM model of the Ducal Palace of Urbino was 
removed from our training set. 

In the second experiment we evaluated the models trained on the TLS point cloud of Fer 
retti Palace. In this case we removed the BIM model from the training set. Since two of the 
selected architectural element classes (column and pillar) are not present in the building, in 
this case we try to recognise the remaining 6 classes. 

The results, reported here in qualitative form (fig. 4), allow us to conclude so far that the 
use of synthetic data can be effective in the automatic segmentation of TLS point clouds. 
Of course, this is only a first step and an encouraging scenario to be explored and analysed 
with other applications to support the ScanToBIM process of historic buildings. 


+ 


Conclusions 


In this article, a method, still in the process of being perfected, oriented towards the instruc- 
tion of artificial intelligence for the segmentation of point clouds has been described: that 
is, an attempt has been made to improve the discernment capacities of neural networks, 
helping them with a form of collective intellisence built from the disciplinary foundations of 
design. These results in themselves constitute an interesting and novel approach, especially if 
we consider the potential for use and re-use of existing models, generated over the last 40 
years or more, in terms of interoperability and sustainability of digitisation; but also in terms 
of formulating the axiom of digital heritage in itself, not as a mere copy. 
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Fig. 4. Qualitative results 
about the segmentation 
of point clouds for the 
Ducal Palace. 


Another interesting food for thought comes from the essay [Clivaz 2020, pp. 67-73] in which 
Robert Wachal's 1971 text is recalled. He raises what he sees as the main problem of the hu- 
manistic approach to computer science: to hope that the time will soon come when human- 
ists will start asking new questions, also referring to artificial intelligence. Clivaz, too, emphasises 
that his ‘personal vision’ is an open appeal worthy of the attention of scholars today. So, also for 
us, CH experts, it is perhaps time to turn the telescope upside down: to start asking artificial 
intelligence new questions, the questions posed with increasing urgency by a fragile heritage. 
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Automated Modelling of Masonry 
Walls: a ML and AR Approach 


Pierpaolo D'Agostino 
Federico Minelli 


Abstract 


A methodology for the automated delineation of brick masonries from images to a vector repre- 
sentation is discussed in this paper Python environment is chosen for the coding activity in order to 
provide automation to the process. Edge detection and vector delineation of brick joints are followed 
by a phase of brick clustering for masonry classification. The implementation of the process is tested 
on a video sequence to simulate an augmented reality application for masonry detection. 


Keywords 
masonry delineation, Canny algorithm, AR, digital survey. 
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Fig. |. Real cases of 
exposed masonry: 
Cellammare Palace 

(on the left) and the 
‘Sferisterio’ in Fuorigrotta 
(on the right) in Naples. 


Introduction 


Parametric and algorithmic modeling are constantly becoming increasingly useful tools for 
the automation of processes destined, among other things, to the management and use of 
cultural heritage, in particular taking into account the digital evolution that the tools for its 
recovery and restoration have had, opening up to the implementation of techniques such as 
machine learning and augmented reality Particularly in the Italian context, in fact, it is known 
how vast is the heritage worthy of attention for its redevelopment and restoration, both for 
historical buildings and for modernist buildings that also have valuable characteristics. Clearly, 
it is in this direction that the new frontiers of digital surveying are evolving by the possibility 
of describing the artifacts through their “digital twins” and by their direct use rather than for 
the realization of classic technical drawings. 

Digital survey is often indicated as a starting point for the creation of structured data 
models. In fact, nowadays their use in reverse engineering processes is stable, but still partly 
subject to manual modeling (according to the interpretation of a human being). 

In this spirit of participation in the automation of such processes, we are working on auto- 
mated workflows for the digitization and interactive use of historical and cultural heritage 
that exploit machine learning applications to obtain cognitive feedback in real time from the 
architectural artefact. To this end, augmented reality applications, running on handheld de- 
vices used as virtual reality interfaces, can mimic the human presence in space and provide 
a computer-generated model, in a virtual space, modeled on the real context, with which 
the users can interact. 

The research in progress, taking advantage of an approach specifically base on scripting, 
concerns the digitization of masonry, a so often required specification that could be very 
complex to reach as automatic outcome, especially in historic buildings where different 
construction techniques can be implemented, to extract adequate information to classify 
and model the underlying structure (fig. |). 
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The delineation based on images or video streams of the masonry, in particular is centered 
on “edge detection” to identify the key areas of the masonry structure: firstly, the regions 
where the joints between the bricks are located. This is a fundamental step for the masonry 
morphology classification process as it allows the segmentation of single bricks. 

To obtain a vector representation of the masonry, in order to achieve a referred 3D mod- 
eling, it is crucial to define horizontal and vertical joints between the bricks, detected and 
delineated from the contours. These operations are pointed to find the intersection points 
between the lines in order to outline the contours of each brick and, then, to evaluate 
the area of each closed boundary used to classify and reconstruct the masonry structure. 
Machine learning techniques come into play in this classification phase and in particular, an 
approach based on data clustering is implemented for the recognition of bricks with faces 
of different dimensions exposed. The methodological approach is described below. 


186 


Fig. 2. Layout of the 
proposed workflow. 


The ability to carry out this operation in real time and in an unsupervised way opens up 
the integration of the model thus obtained with augmented reality systems. Multi-channel- 
ing through the use of augmented reality thus finds an interesting development path with 
high added value since the quality and quantity of data acquired and processed to support 
the interaction enrich the experience of interaction with the monument by placing it in an 
ideal context: real-time feedback of the architecture. The fundamental innovation, in fact, 
regardless of the scale dimensions of the displays (tablets, smartphones) is the activation of 
the development of an integrated interaction project which, by exploiting augmented reality, 
enhances communication and knowledge, managing to simplify and make the interaction 
with architecture more complete and engaging, raising the level of attention (fig. 2). 
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Methodology 


What we would describe here is a workflow for automatic detection and modeling of pe- 
riodic walls. We started by considering how to apply parametric and algorithmic modeling 
in such automation processes used to take. 

The possibility in objective processes that allow automatically to pass from the point cloud 
to parametric models is still very rare, for example in Building Information Modeling ap- 
proaches, for which the possibility of identifying, for example, openings or automatically 
segmenting building components is still an arduous challenge. Or; for example, in the models 
intended for the detailed analysis of its parts (for example FEM analysis), of those character- 
istics that make up the individual construction components. 

Starting from simulated data but corresponding to possible applications starting from real 
data, the process leads to the production of solid models of brick walls. The theoretical 
approach aims to be applied in several city models applications. Procedural city modeling 
ranges from the digital reconstruction of entire neighborhoods to the production of single 
building models [Zhu et al. 2016]. From the analysis of available studies on digital reconstruc- 
tion of urban areas using algorithmic procedures [Musialski et al. 2013] and automated as— 
built modeling [Patraucean et al. 2015] our modeling approach can be evaluated with these 
methods. The usage of images and point clouds for the recognition and modeling of archi- 
tectural features are of great interest and in continuous development. The automation of 
this parametric architecture modeling process from on-site survey materials is also a very 
active field in scientific research today [Czerniawski and Leite 2020]. However, differently by 
standardized situations, the morphology of building walls can be very complex, especially in 
historic buildings, but several techniques can be implemented to extract adequate informa- 
tion to classify and model the underlying structure. The image-based delineation of mason- 
ry, in particular relies on edge detection to identify key areas of the masonry structure: first, 
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Fig. 3. The masonry 
classification process: 
brick face occurences 
detection to define the 
wall textures. 


the regions where the gaps between the bricks are located. This is a fundamental step for 
the masonry morphology classification process as it allows the segmentation of single bricks, 
but the subsequent 3D modeling phase also requires an investigation as it allows to obtain 
a full reconstruction of the asset [D'Agostino & Minelli 2020]. 

Moreover, this work focuses on the automatic detection of masonry from images and its 
interaction in AR applications. To answer to the goal of optimization of the ongoing inves- 
tigation in this connection between AR acquiring process and a future-proof real time 
processing, with the need to test several outcomes, an image-based approach to vectoriza- 
tion of wall textures characterized by horizontal rows is delineated and tested on different 
masonry samples. The programming language Python is selected, for its flexibility and effec- 
tiveness for code writing activities in order to achieve process automation, for the coding 
activities to achieve the automation of the process. 

The work specifically aims to digitally reconstruct the masonry walls based exclusively on 
rectified images. As anticipated, the approach proposed first seeks to achieve a vector 
representation of each brick in respect of the real masonry texture. The second purpose, 
rarely addressed in the literature before, is to create an interactive delineation of masonry 
on the basis of the current arrangement of the bricks. The approach tries to be effective on 
masonry made by bricks of a single size. 

Segmentation and three-dimensionality for the reconstruction of the individual bricks is 
tested on several images of masonry to verify the consistency of the proposed workflow 
to the unpredictable conditions that can occur when dealing with real masonry. The RGB 
source images are transformed into the HSV color representation in the Canny algorithm 
for the first brick edge detection. The binary image of the edges is used to extract the con- 
tours of the brick. In order to obtain a vector representation of the masonry, the UV coor 
dinates of the edges of the horizontal brick joints are first detected and delineated by the 
contours. The vertical joints are detected in a second step and added to the previous one, 
respecting the height of each row of bricks. The intersection points between the lines allow 
to outline the contours of each brick. The area value of each closed contour is calculated to 
classify and reconstruct the masonry structure. 

Mean Shift algorithm is used to sort the outlines of the front bricks by the outlines of 
the side facing ones. The quantity of clusters detected and the number of occurrences of 
front and side bricks allows to classify each masonry analyzed in one of the 3 categories 
investigated in this work (fig. 3). Finally, a reconstruction of the 3D mesh of the masonry is 
performed and the geometric model is saved in a DXF file. 

A data clustering approach is implemented for recognizing bricks with faces of different sizes 
exposed. The Mean Shift clustering algorithm is then used to separate the oriented bricks with 
the front face in view from the side ones, based on the area value of their respective bound- 
aries. The front facing bricks, in fact, show a lower value of the boundary area, while the side 
facing bricks are characterized by a greater value of the boundary area. Choosing the Mean 
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Shift clustering algorithm relies on its ability to directly estimate the number of clusters from 
the dataset. In fact, other clustering algorithms require the user to manually specify the cluster 
number file to search for in the dataset. The masonry investigated in this study, however, range 
from types in which a single cluster of bricks is found to types in which two clusters occur 
Deepening the possibility to join AR visual acquisition with the described workflow, we 
propose that the vectorization of walls’ joints could pass for their identification directly on 
visual flows. So, the approach studied for raster images is applied to video sequences in 
order to allow a real time detection and vectorization of the masonry.An augmented reality 
application is therefore simulated on a video stream, acquired in real circumstances (fig. in 
cover page). An effective delineation of horizontal rows is obtained with robust outcomes 
on several frames of the video stream. This opens to the complete vectorization of the 
masonry in real time, that will be addressed in future studies. 


Conclusions 


Multi-channeling through the use of augmented reality thus finds an interesting develop- 
ment path with high added value because the quality and quantity of data designed to sup- 
port the interaction enrich the experience by placing it in an ideal context: when needed 
and to whom really need. 

The fundamental issue, in fact, is not so many the scale dimensions of the displays (mega— 
screen, tablet, installation, smartphone) that make the difference, as the development of an 
integrated communication project which, by exploiting augmented reality, enhances com- 
munication by managing to simplify and make the interaction more complete and engaging, 
raising the level of attention. 
Segmentation and three-dimensionality applied in wall digitation is tested on a video of the 
masonry to verify the consistency of the proposed workflow to the unpredictable condi- 
tions that can occur when dealing with real masonry. 
The results obtained in the study can be applied in architecture surveys and fruition application 
to establish a direct connection between the captured image and the reconstructed geometry. 
Practical applications of this procedure can also be found in the automated BIM modeling 
process for 3D reconstruction of buildings from survey material and in the automated FEM 
modeling of masonry structures, as it can provide a detailed model of masonry morphology. 

Future work should consist of further experimentation on different and more complex ma- 
sonry structures to obtain results of wider applicability. The masonry classification, already 
attained with images, should be also tested on video sequences. 
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Data Modelling in Architecture: Digital 
Architectural Representations 


Elisabetta Caterina Giovannini 


Abstract 


Digital Architectural Representations represent the most fruitful field of research of the last decade. 
Digital technologies and the use of internet in architectural representation shows how 3D visualization 
combined with storytelling can help to spread scientific knowledge over the web. These new technol- 
ogies also affect the way of thinking 3D models, how to design them and how to build their related 
knowledge with the purpose of future reuse of information and data. 

The paper is focused on the analysis of current methodologies and workflows for data modelling in 
Architecture to better understand the potential of using standards in the 3D modelling sector with a 
focus on cultural and architectural heritage. 


Keywords 
3D models, semantics, data models, standards, ontologies. 
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The Complexity of Digital Architectural Representations 
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geometric modelling, the presence of diverse data and documents prefigure the need 
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uction process, we can see 
ualitative and quantitative values of 


the available documentation. This type of data and information support both geometric 


and informative modelli 


ng, considered as two indivisible and inter—related 


components 


of the same process. An example of that is the common practice of using the semantic 


architectural structure 
proper of the architect 


+ 


to digitally create the parts of a 3D model according to logics 


ural field and to us 


e digital architectural elements as reference 
objects to connect additional information [De Luca 2013; G 


of knowledge production. This assumption is valid not only fo 
all human objects of production that can be manuscripts, sketches, drawings, maquettes, 
etc. These pre-existing data can then be used for information modelling and three- 
dimensional modelling enriching diverse Levels of Knowledge ( 


Knowledge Representation in Architecture 


iovannini 2017; Quattrini, 
Battini, Mammoli 2018]. A digital architectural representation can 
a visual and graphical expression of an interpretative activity and 


be, then, considered as 
a constitutive element 
D models but also for 


LoK). 


Applications of Artificial Intelligence (Al) to Cultural Heritage (CH) have been developed 
with a varying fortune to produce innovative tools for documenting, managing, and visiting 
cultural heritage. From the representation of cultural history, digital semantic archives, 


tools to support visitor's interpretation, augmented reality and robotics, the app 


ication of 


Al has been applied to the whole humanistic area. In the Architectural Heritage field, Al 


is commonly used for storytelling, restoration analysis and 3 
also used to develop ontologies [1] to allow computers to perform automated 
about data and information all over the world. Software 


+ 


D model classification. Al is 


reasoning 


Engineering, on the other hand, 
started to use conceptual modelling as a representation of a system to describe 
Tools for designing and creation of online visualization of data, according to the 


concepts. 
rules that 


govern the web in the past, and more recently the semantic-web cannot avoid the use 


of Information model 


ing to manage and structure data and information. In the case of 


digital architectural representations, the text analysis and the source where architecture 
is represented in a bi-dimensional way are enriched by three-dimensional information 
derived from the digital acquisition or three-dimensional modelling. The recent need 
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Fig. |. Conceptual data 


modelling overview. 


On the top, conceptual 


modelling of H-BI 
object classes using 


CRMba extension. On 
the top-right, resource— 


based modelling 


processes using CHARM 
model. On the bottom— 
right a conceptual model 


for the reality-based 


3D modelling process 
using CRMdig extension. 
On the left, conceptual 


representation of 
knowledge in the 
Architectural field. 


for interoperable processes that characterize most of the research on documentation 
and representation of architectural heritage has emphasized the occurrence of various 
approaches. Some studies analyze conceptual modelling to define and reorganize the 
information and material available for the comprehensive use of a digital asset. The theme 
of processes in digital modelling is useful to trace choices, decisions on three—dimensional 
models using visualization codes [De Kramer 2020; Giovannini 2020; Apollonio, Gaiani, 
Sun 2013]. Declaring the accuracy or reliability of 3D models [Apollonio et al. 2017; 
Bianchini, Nicastro 2018] including those obtained using tools and algorithms (for 
example in the case of digital survey or photogrammetric acquisition) is a practice 
mostly used in approaches for geomatics and Building Information Modelling applied 
to Heritage (H-BIM) [Maiezza 2019; Garagnani 2013; Quattrini, Pierdicca, Morbidoni 
2017]. Resource-based 3D modelling considered as interpretative process, differs from 
digital acquisition where reality-based data can be considered as formal derivation of the 
original object. Both modelling processes are in relation since the 3D reality-based data, 
if present, affects the modelling and validation process of the derived resource-based 
model. Standards and models for information modelling including conceptual ones have 
long been in use in the cultural heritage sector: the CIDOC Conceptual Reference Model 
(CRM) standard [2], the ISO 21127:2014 also known as CIDOC-CRM , often associated 
with the controlled vocabulary of the Getty Institute (AAT) [3], is the most used ones. 
Recent trends demonstrate that ontologies and conceptual models are not that different, 
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and that combined can allow standardising the documentation of cultural heritage. In 
the archaeological field the Cultural Heritage Abstract Reference Model (CHARM) 
[4] is an ontology for cultural heritage expressed in Conceptual Modelling Language 
(ConML) [Gonzalez-Perez et al. 2012].The Extended Matrix (EM) [5] is a visual language 
of knowledge representation in the field of virtual reconstructions with a stratigraphic 
approach [Demetrescu 2015]. In the architecture, engineering and construction (AEC) 
industry, the reference standard is the Industry Foundation Classes (IFC) data model [6], 
a metadata schema capable of describing architectural semantics and making Building 
Information Modelling (BIM) models interoperable between different software solutions. 
The IFC guarantees the management of geometry but it does not allow the addition of 
customized information outside the context of the construction industry. Nevertheless, 
some emerging research proposes an IFC classification for architectural heritage asset 
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[Diara, Rinaudo 2020] or an architectural heritage semantic 3D documentation for 
the reusability of 3D city models [Noardo 2018]. To establish a dialogue between the 
architectural field and the cultural heritage assets a recent study proposes a conceptual 
model based on the CIDOC-CRM standard to describe a building in its parts, as encoded 
by a BIM software using the CIDOC-CRMba extension [7], developed to describe built 
archaeology [Parisi, Lo Turco, Giovannini 2019]. With that model, it was possible to 
describe the morphological elements that characterize a building but it fails in describing 
the link between the geometric parts and their spatial coordinates. The conclusion was 
partially acceptable if we think that the CIDOC-CRM was born to describe assets about 
museum collections and not about architecture. 


Conclusions 


Considering the diverse research conducted, the possibility of creating conceptual 
models capable of managing three—dimensional data and descriptive metadata on the 
documentation of architectural heritage is still missing. The CRM-dig [8], a CIDOC- 
CRM extension, is a model capable to manage the complexity of the reality-based data 
acquisition, but it does not clarify and explain the relationships between the source used, 
the data extracted from it and its use for geometrical modelling. The IFC, on the other 
hand, can be used to describe geometric information of BIM or H-BIM models. The 
challenge is to create an efficient data model that allows semantic traceability of data. 
A novel semantic organization of data is necessary for the development, of platforms, 
analysis tools and algorithms able to manage structured data, make queries for different 
purposes in complementary disciplinary domains emphasising their combined potential. 
There is a need fora common conceptual model that reflects the complexity of the three— 
dimensional modelling process. Conceptual modelling should focus on the representation 
of architecture in all its forms (drawings, surveys, digital models) and should represent 
both digital and physical properties. A conceptual reference model for the digital 
representation of architecture (fig. 1) should first identify the architectural evidence, built, 
or only represented ones, the parts of which it can be composed and how these can be 
digitally represented. The knowledge about an architectural asset is also composed of a 
set of resources that also need to be digitized and that contribute to the creation of the 
“architectural” digital asset. Then the relationship between digitized resources and three— 
dimensional modelling can take place by mapping diverse interpretative and modelling 
processes creating different levels of knowledge. The knowledge produced, can then be 
used by digital tools able to read the conceptual grammar of the asset: an information 
system in which three-dimensional models and historical documentation is collected 
and organized. To reuse data, data models are necessary and even if they do not follow 
standards, they must at least be stated because this is how computational technologies 
and machine can see the human world. 


Notes 

[1] Ontology is the theory and the Information Model is the application. Information modelling is here intended as "a 
representation of concepts, relationships, constraints, rules, and operations to specify data semantics for a chosen domain 
of discourse. The advantage of using an information model is that it can provide sharable, stable, and organized structure of 
information requirements for the domain context.” [Lee 1999] 

[2] The CIDOC Conceptual Reference Model (CRM). http//www.cidoc-crm.org/ (March 202 1) 

[3] Art & Architecture Thesaurus (AAT) — Getty Research Institute. https//wwwgetty.edu/research/tools/vocabularies/aat/ (March 
2021) 

[4] Cultural Heritage Abstract Reference Model (CHARM) — INCIPIT, CSIC. www.charminfo.org (March 202 1) 


cel 


] The Extended Matrix formal language for virtual reconstruction processes. http://osiris.itabc.cnrit/extendedmatrix/ (March 202 1) 


EN 


] Industry Foundation Classes. https://www.buildingsmartorg/standards/bsi-standards/industry-foundation-classes/ (March 2021) 


[7] An extension of CIDOC CRM to support buildings archaeology documentation. http://www.cidoc-crm.org/crmba/ (March 2021) 


[8] An extension of CIDOC CRM to encode “provenance” of digitization products. http//www.cidoc-crm.org/crmdig/ (March 2021) 
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REPRESENTATION CHALLENGES 
Augmented Reality and Artificial Intelligence in 
Cultural Heritage and Innovative Design Domain 


Image-Based Modelling Restitution: 
Pipeline for Accuracy Optimisation 


Marco Limongiello 
Lucas Matias Gujski 


Abstract 


The paper presents an innovative approach to support survey methods by applying Al algorithms 
to improve the accuracy of point clouds generated from UAV imagery. The work analyses different 
photogrammetric accuracy parameters in a first step, such as reprojection error and the intersection 
angle between homologous rays, verifying that a single parameter is enough to evaluate the accuracy 
of the photogrammetric restitution. Therefore, some of the calculated parameters were analysed 
through a Self-Organizing Map (SOM) to reach a compromise between the value of the variables 
analysed and the noise reduction associated with the 3D model definition. In the case study, it has 
been observed that the parameter that most influences the noise in the photogrammetric point 
clouds is the intersection angle. 


Keywords 
UAV, photogrammetry, accuracy, point cloud, SOM. 
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Developed Methodology and Quality Features 
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*SOM - Self-organizing map [Kohonen 2001]: is an artificial neural network machine 
learning technique [Teruggi 2020, pp. 1-27] usually used for visualization and analysis 
of high-dimensional data. Moreover, SOM is used for data clustering. Self-organizing 
maps can be combined with dimensionality reduction methods as multi-dimensional 
scaling [Kurasova 2011, pp. 115-134]. The number of clusters to be brought into the 
accuracy analysis is extracted from the graphical representation of the dendrogram. A 
dendrogram (in first page) is a diagram representing a tree that shows the hierarchical 
relationship between object and used to visualise the similarity in the clustering process. 
In clustering techniques, the dendrogram is used to provide a graphic representation. 


Analysis of Individual Accuracy Parameters 


A very noisy standard portion, 50 cm wide, contai 
analysed for the purpose of the st 
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distribution is the one best-fitting the interpolated data, therefore it was chos 
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It is possible to observe (fig. 1) how the filtering of the point cloud by analysing the RI 
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statistical approach does not generate a good degree of filtering for the section under consid- 
eration; in fact, the procedure removes some spots mostly scattered, but it does not lead to 
great advantages in noise reduction. However most of the isolated points were not filtered out. 
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Fig, |. Single parameter 
analysis of RE and 
intersection Angle. 


A better result for noise reduction is obtained by filtering the point cloud according to 
the average angle of intersection, calculated for each Tie-Point, and then analysing the 
acquisition geometry. 

Excluding small average angles of intersection, we have obtained surfaces that are much 
more realistic and less noisy. Pushing the filter too much, setting high angle values as thresh- 
old, compromises the amount of data necessary for the representation; in fact, by setting 
an average angle over 20° as a threshold, large quantities of points belonging to the vertical 
walls are removed (fig. |). 

As for the parameter of the average distance between the i-th camera and the TPi, it is a 
parameter that does not affect the noise. It was also considered to take into account the 
density of the point cloud: high distances do not allow high GSDs, and therefore not very 


dense point clouds and cloud sections. 


SOM Analysis and Conclusion 


It was decided to experiment with a SOM method in order to take into account not only 
a single accuracy parameter but all the measured parameters at the same time. Using the 
Matlab Neural Network Clustering App, 4 values was used as input, which were the RE, 
number of images, average angle and average distance. The SOM Layer loaded with an 8x8 
network and 100 epochs. The graphical representation of the dendrogram is used to select 
the number of clusters for the accuracy analysis. Once the dendrogram is created, 7 cate- 
gories can be identified. It was decided to divide these 7 categories into 4 groups (fig. 2): 3 
accuracy categories (High, Medium and Low) and a noise category. In order to reorganize 
the clustering of these accuracy levels, we analysed the average angle, the parameters of the 
maximum and minimum angles, and the number of cameras. 

The established clusters 3 and 4, which have the widest mean angle, maximum angle and 
minimum angle, are classified in this new classification as the High Class. Cluster 2 has been 
set as the Medium Class, clusters | and 7 as the Low Class, and finally clusters 5 and 6 as 
the Noise Class, with lower intersection values. 

Analysing the cluster of the point cloud we overlapped (fig. 2) the ‘High Layer Accuracy’ 
containing the highest angle intersection with the ‘Noise Cluster’ containing the lowest val- 
ue. It can be seen that the ‘High Layer Accuracy’ turns out to be the best fitting point set for 
vertical walls and ground. 
The work shows that filtering the point cloud by evaluating the RE and using a statistical 
approach individually does not produce good filtering quality; in fact, even with high percen- 
tiles, some outliers were still not filtered out. Filtering according to the average intersection 
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Fig, 2. Clustering 
and accuracy classes 
extracted. 
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angle parameter alone produces a better result for ‘noise’ reduction. However, filtering 
through high angles can compromise the data density and cause, as a result, loss of informa- 
tion. With the SOM approach, using all the parameters calculated at the same time and the 
clustering process, the value ofthe RE does not change significantly. By analysing the num- 
ber of images forming the cluster, the greater the number, the greater the noise reduction 
obtained. Using the cluster angle analysis, we can conclude that the clusters with the highest 
base-to-height ratio are considered to have the highest noise reduction. Generally, a trend 
can be defined, i.e. clusters with higher mean angles generate TP sections with less noise. 
From the analysis above, the parameter that most influences the noise in a TP point cloud 
is the mean intersection angle. Therefore, we can conclude that the advantages of using Al, 
in particular SOM, a relatively simple method applicable to point clouds, has allowed a fast 
clusterization, from which TPs can be selected with different accuracies, depending on the 


subsequent purposes. 
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REPRESENTATION CHALLENGES 
Augmented Reality and Artificial Intelligence in 
Cultural Heritage and Innovative Design Domain 


From Al to H—BIM: New Interpretative 
Scenarios in Data Processing 


Federica Maietti 
Marco Medici 
Ernesto ladanza 


Abstract 


The paper results from preliminary research experiences focused on the use of Artificial Intelligence 
as a tool for processing the large amount of data that can be obtained from digitisation processes ap- 
plied to the Architectural Heritage. The new interpretative scenarios will be outlined starting, on one 
hand, from a series of consolidated experiences in the field of three-dimensional survey, modelling 
and semantic enrichment, and, on the other from the use of Augmented Reality tools for the fruition 
of the Heritage itself. 

The research aims to further investigate the great potential of processing point cloud models using 
Artificial Intelligence, to extrapolate, from the digitized data, information levels that go beyond shapes, 
offering better integration within the Building Information Modeling environment. 


Keywords 
cultural heritage, point cloud, artificial intelligence, H-BIM, data management. 
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Research Scenario 


Today's increasingly fast digital surveying tools boost the speed of scanning, but also the 
amount of data captured. This trend is leading to the development of research avenues 
in which Artificial Intelligence (Al) is aimed at the segmentation and discretization of data, 
mainly, but not only, for the recognition of shapes (object detection) and structures [Grilli, 
Menna, Remondino 2017]. This procedure is triggering a debate, still unresolved, that in- 
volves the interpretative aspects of the uniqueness that characterises Cultural Heritage, 
while the need to trace specific directions emerges more and more. 
The research therefore aims to further investigate the great potential of processing point 
cloud models using Al towards data integration in Building Information Modeling (BIM) en 
vironment applied to Heritage. 
The definition of a methodology able to automatically recognise specific characteristics 
from 3D point clouds can lead to the definition of new data sets, aimed at documentation, 
conservation and restoration, which can enrich BIM models, also thanks to a now necessary 
inclusion of advanced features in the Industry Foundation Classes (IFC) standard through 
new and shared Property Sets. 

The integration of advanced surveying techniques, Machine Learning (ML) and Deep Learn- 
ing (DL) in new standards for Heritage information modelling (H-BIM) can have a great 
impact on the process of documentation, representation, analysis and interpretation of Her- 
itage [Bienvenido—Huertas et al. 2019], creating new representation levels and application 
scenarios in Heritage management, conservation and restoration. 

This research scenario is strongly connected to an additional data management level, related 


to the application of Augmented Reality too 
of experiments were carried out in order to 
BIM environment within an open standard we 


of Heritage buildings also for conservative pu 
n this direction, the research includes the use 


+ 


s aimed at ‘onsite’ data exploitation. A series 
create semantically enriched digital models in 
b platform [|]. Data access to the platform via 


Augmented Reality applications allows Heritage fruition, analysis, monitoring and assessment 


rposes. 
of semi-automatically processed data, related 


o the state of conservation and technical assessment for asset management, maintenance 


and decision—making purposes by using mobi 


e devices. 


Digital Data Processing for Heritage Conservation 


The need to document Cultural Heritage — c 
has led to an increasingly widespread use of 3 
are able to produce very accurate models in 


survey, the advantages of speed and metric a 


data can be very time-consuming and comp 


haracterized by uniqueness and complexity — 
D surveying technologies. These technologies 
a very short time. While, during the on-site 
ccuracy are evident, the processing of these 
ex. Today, the trend is to produce even fast- 


er and more robotic instruments (mobile devices) [Gallozzi, Senatore, Strollo 2019] that 
make it possible to scan (especially indoor environments) in a single shot. This produces an 
increase in scanning speed, but also a further increase in the amount of surveyed data. This 
course is leading to the development of a whole Artificial Intelligence line of research aimed 
at segmenting and discretizing data. 
A further layer to be added to this framework is related to digital data representation and 
management. BIM is considered today the latest frontier of three-dimensional architectural 
representation, design and management of digital data, where interoperability [Osello et. 
al. 2015] is one of the central attributes. BIM software applications are growing rapidly and 
they are also becoming increasingly essential in conservation and restoration applications 
[Chiabrando, Lo Turco, Rinaudo 2017], thanks to their ability to integrate different informa- 
tion and features in relation to the model geometric shapes [Apollonio, Gaiani, Sun 2017]. 
However, the IFC, the interoperable standard for BIM, currently lacks in providing a solution 
for managing the preservation of the Architectural Heritage. Moreover, unlike geometric 
characteristics [Murtiyoso, Grussenmeyer 2019], an automatic process does not uniquely 
determine surface features. They become consistent and significant only if critically inter 
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Fig. |. Analysis of surface 
specifications by means 
of point cloud processing 
and data management in 
BIM environment. 


preted. Therefore, only if performed with a systematic and well-documented methodology, 
surface specification analysis can allow a completely non-invasive and strongly helpful sup- 
port for condition assessment (fig. |). 

Currently, the interpretation of this data and its visual representation as a mapping of the state 
of conservation requires a manual or semi-automatic and rather lengthy process. 


Even if the concept of setting up automatic procedures and automatisms is very tricky in 
the field of cultural Heritage (where each building is unique and requires case by case as- 
sessments), Al technologies will be more and more necessary since the elaboration of huge 
amount of data is one of the most important tasks in the digital era [Jankovié 2020]. These 
processes can allow — starting from a massive set of data — to explore new forms of discretiza- 
tion and classification. From a single historical building or heritage site, it is possible to extract a 
huge amount of data that need Al, Machine Learning (ML) and Deep Learning (DL) processes 
to be analysed and compared. By using point clouds obtained by the 3D survey of a historic 
building, it is possible to process specific set of points (e.g. related to a certain surface) and to 
visualize specific surface features [Grilli, Ozdemir, Remondino 2019]. AI, ML and DL can make 
this process faster and more effective [Malinverni et. al. 2019]. Of course, several parameters 
need to be assessed before the process starts. This procedure can be integrated by exploiting 
these additional layers of information derived from data capturing procedures for automati- 
cally populating the H-BIM model, a research field where there is large room for innovation 
[López et. al. 2018]. The definition of an extended data schema including information from the 
restoration discipline, reflecting a shared vision and approach. 


From Artificial Intelligence to H-BIM 


According to the research scenario about digital data processing for Heritage conservation 
previously outlined, the research aims to establish a new process of point cloud analysis by 
applying Al, ML and DL processes, generating interpretative algorithms allowing the segmen- 
tation and classification of large amounts of data, in order to outline and describe historical 
surface specifications. This leads to a methodological procedure as a connection between 
Al and the automatic recognition of surface specifications to the BIM model, allowing the 
creation of a new data schema for restoration, including in the IFC standard the detailed 
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Fig. 2. Graphical schema 
summarizing the overall 
process. 


documentation of the state of conse 
the application of the interpretative 
IFC by the creation of a new set of' 


rvation directly extracted from the 3D point cloud by 
algorithms. Such properties will be included into the 
abels' able to describe different data and information 


related to Heritage buildings within the BIM process. Specifically, the process (fig. 2) involves 
the selection of databases, the identification of features or properties to be processed 
through automatic recognition, data processing to identify the algorithm capable of recog- 
nizing specific properties, the extrapolation of datasets according to homogeneous features, 
implementation of new datasets in the BIM environment, and the definition of new IFC 
‘translators’. The careful selection of interpretation criteria and parameters will be an essen- 
tial part of the workflow described above. 
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This overall process is closely linked with the development of semantic models, beginning with 
the increasingly advanced identification of features to be incorporated into the BIM models, 
thanks to Artificial Intelligence. Via Virtual Reality applications, these models can be the foun- 
dation for advanced explorations, by leveraging a collection of features relevant to the state 
of conservation, materials, previous initiatives, technological documentation, directly on-site. 


Conclusion 


The outlined methodology is the first step in a starting research process, which requires 
consolidating the described procedures through a number of data sets. Anyway, the project 
results may have impacts by reaching interesting improvements in some challenging steps of 
Heritage digitization and data processing for conservation and restoration. 

The state of the art outlined on the basis of national and international research related to 
the application of Al to Cultural Heritage shows indeed an articulated panorama composed 
of image classification algorithms, point cloud segmentation and representation models able 


+ 


to estimate levels of intervention on historical buildings. However, there is large room for 
experimentation and many unsolved issues that make the scenario still open and require 
that multiple levels of knowledge of the Heritage derived from digital languages and tools 


find a common ground. 
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At European level, several projects and initiatives are developing the use of BIM for regen- 
eration, but there is still room for progress in the field of application to cultural Heritage. 
Through the described methodology, geometries and shapes can be integrated with differ- 
ent information regarding materials, state of conservation, historical documentation, previ- 
ous restoration works, etc. 

Al can lead to the development of new, increasingly targeted segmentation algorithms, also 
to trigger new uses and re-uses of digitised Heritage. 

Future research scenarios foresee an integration of Computer Vision and Augmented Re- 
ality in the process, for ‘onsite’ applications, exploiting the data extrapolated through the 
described procedure, for the enhancement and management of Cultural Heritage, for mon- 
itoring or architectural restoration project. 


Notes 


[|] These experimentations have been developed under the Horizon 2020 project “INCEPTION — Inclusive Cultural Heritage 
through 3D semantic modeling”, funded by the European Commission in 2015 and concluded in 2019 (GA 665220). The 
project, leaded by the Department of Architecture of the University of Ferrara, focused on semantic modelling of Cultural 
Heritage buildings using BIM to be managed through the INCEPTION web based platform for advanced deployment and 
valorisation of enriched 3D models, towards a better knowledge sharing and enhancement of European Heritage. 
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Augmented Reality and Artificial Intelligence in 
Cultural Heritage and Innovative Design Domain 


Machine Learning for Cultural Heritage 
Classification 


Michele Russo 
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Abstract 


Cultural Heritage (CH) assets may be defined as integrated spatial systems composed of inter- 
connected shapes. The classification and organization of geometries within a hierarchical system 
are functional to their correct interpretation, which is often performed using 3D point clouds. The 
recurring shapes recognition becomes a crucial activity, nowadays accelerated by Machine Learning 
(ML) procedures able to associate semantic meaning to geometric data. An interdisciplinary research 
team [|] has developed a ML supervised approach, tested on the Milan Cathedral and Pomposa 
Abbey datasets, which presents an innovative multi-level and multi-resolution classification (MLMR) 
process. The methodology improves the learning activity and optimizes the 3D classification by a 
hierarchical concept. 


Keywords 
machine learning, cultural heritage, multi-resolution, hierarchical 3D classification, level of detail. 
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Fig. |. External and 
internal photos of the 
Milan Cathedral and 
Pomposa Abbey, with 
details of the monumental 
capitals of the Cathedral 
and the wooden roofs 

of the Abbey (authors’ 
images). 


Introduction 


Cultural Heritage (CH) assets are complex artifacts whose knowledge passes through analyz- 
ing an integrated system of forms interconnected by dependence or proximity relationships. 
The recognition and classification of 3D data become essential to (re)assign a hierarchical 
and functional meaning to acquired point clouds. The manual classification activity, which is 
very time-consuming, can be nowadays replaced by an automatic one based on Artificial 
Intelligence (Al) approaches, such as Machine Learning (ML) or Deep Learning (DL) methods. 
These Al approaches have many bottlenecks in the CH field, mainly due to the complexity and 
variability of the shapes, the reliability of the interpreted data, the scalability of the process and, 
often, the absence of annotated data. In this paper, a supervised ML method applied to CH 
is introduced and evaluated. It is based on a Multi-Level Multi-Resolution (MLMR) approach, 
which considers the various geometric details present in the point cloud. Two complex 3D 
datasets related to Milan Cathedral and Pomposa Abbey are processed to test the developed 
methodology and demonstrate its flexibility and efficiency with different scenarios. 


State of the Art 


Several investigations performed to classify (or semantically segment) 3D point clouds in 
the architectural heritage field using automatic ML and DL methods. Grilli et al. [2018, pp. 
1-8] presented a supervised ML approach to transfer classification data from 2D textures 
to 3D models, whereas Grilli et al. [2020] used a Random Forest (RF) classifier with geo- 
metric features to derive architectural classes from point clouds. In the DL domain, Pierdicca 
et al. [2020] trained the ArCH dataset (http://archdataset.polito.it/) with a Dynamic Graph 
Convolutional Neural Network (DGCNN) using meaningful features (colour, normals, and 
HSV), providing promising results.A comparison of ML and DL techniques for the classifica- 
tion of architectural point clouds [Matrone et al. 2020,] shows that similar accuracy results 
can be achieved. However, ML requires much less time and does not need large 3D data- 
sets in the training phase. For this reason, we hereafter present a supervised ML approach 
adapted to the different geometric levels of detail and architectural classes. 


The Case Studies and Classification Purposes 
Two datasets, with different dimensional and morphological characteristics but presenting 


similar architectural elements, were selected for validating the methodology. The first case 
study is the Milan Cathedral (fig. 1) which was digitally recorded in the last decade with 
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Fig. 2.A view on the 
point clouds of the 
two datasets: The Milan 
Cathedral (left) and the 
Pomposa Abbey (right). 


Fig. 3. Schema of the 
MLMR iterative process. 


several integrated acquisition campaigns to generate parametric models [Fassi et al. 201 |, 
pp. 462-487], and define a complete 3D point cloud (fig. 2) at a uniform average resolu- 
tion of 5 mm [Achille et al. 2020, pp. 331-341]. The classified point cloud may facilitate the 
3D data exploration, allowing the integration between archival sources and surveyed data 
on a web-based BIM-type platform, which can be consulted in situ or remotely. This data 
organization can also allow multi-scale planning and implementation of conservation and 
management projects and the quick extraction of 2D representations already classified. The 
second case study is the Pomposa Abbey (fig. |) surveyed in 2014 to generate a complete 
3D dataset (fig. 2) at a uniform average resolution of 2 cm [Russo et al. 2014, pp. 305-312]. 
In this scenario, the 3D classification activity can foster access to the system’s knowledge, 
supporting its graphic restitution and the monitoring activities at different scales. Besides, it 
can facilitate the “quantification” of the building, collecting helpful information for planning a 
conservation intervention and evaluating the transformations over time. 


The Methodological Workflow 


The high level of complexity of the case studies highlights two different bottlenecks: on the 
one hand, the processing of massive datasets cannot be simplified unless losing the level of 
detail useful in the element recognition. On the other hand, the high number of semantic 
classes raises the management complexity and reduces their identification accuracy [Teruggi 
et al. 2020]. An iterative methodology [Grilli et al. 2020] has been developed to overcome 
these bottlenecks, classifying 3D data in multiple steps according to their information levels 
(fig. 3). The proposed hierarchical structure is referred to the data density, the morphological 
and compositional complexity, and the classification purpose. At each level of detail (LOD), 
the workflow foresees two working steps: 

1) The selection of ‘covariance features’ [Blomey et al. 2014] extracted within specific spheri- 
cal radii, for the automatic recognition of local geometric characteristics of 3D datum. 

2) A small manual annotation to train a Random Forest algorithm [Breiman 2001, pp. 5-32], 
associating each portion identified by the features to architectural meanings. 

The training dataset's selection evaluates the presence of the elements to be classified. 
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Table |.Timing and metric 
summary for the two 
datasets according to the 
three classification levels. 
*) 18 Core Processor; 
(**) 12 Core Processor. 


Fig. 4. The classification 
process ofthe Cathedral 
(top) training portion 
and the Basilica (bottom) 
according to the 
resolutions and levels 

of detail sought by the 
features. 


Experimentation and Results 


The classification process refers to the following three—level of details (fig. 4): 

— In the first level (LI), a point cloud subsampled at 5 cm, with min/max radius of the fea- 
tures between 20 cm and 2.5 m, was processed, subdividing the churches into architectural 
macro—categories; 

— In the second level (L2), after transferring the LI classification to the 2 cm resolution 
point cloud, features extracted with radii between 10 cm and | m were used to split the 
architectural elements into macro-elements; 

— In the third level (L3), receiving the L2 subdivision, features with radii of 0.5 and 5 cm 
were used on the 3D point cloud with a 5 cm density for the Cathedral and 2 cm for 
the Abbey This allowed identifying the single architectural monolithic and technologically 
coherent components. 

Both the processing time and the metrics commonly used in ML to define reliability of the 
results (“Precision, “Recall,” and “Fl score” [Goutte et al. 2005, pp. 345-359]), were analyzed 
to evaluate the classification performance (tab. 1). 


F1 Score (%) 


The achieved results highlight the importance of using point clouds with a level of detail (geo- 
metric resolution) and density suitable to support subsampling or backward interpolation pro- 
cesses consistent with identifying architectural elements. Moreover, if the features radii affect 
only the time in shapes research and the complexity of the architectural connections affects 
just the classification process, the geometrical density and the processor capacities affect the 
whole timing workflow (tab. |). The reported quality metrics show the possibility of obtaining 
excellent results quickly, identifying even very complex geometric structures. 


Conclusions and Future Research 


In this paper, a new iterative strategy for supervised automatic ML classification of 3D point 
clouds of complex Cultural Heritage is presented. Few annotated 3D data were necessary 
and very detailed semantic segmentation results could be achieved. The cognitive contribu- 
tion in the supervision phase is crucial in the correct definition of classes and the choice of 
training and validation sets. These steps are also critical to adapt the general process to the 
specific case study and different purposes. 

In the future, the relationship between classification levels, cloud resolution, and feature 
search radii will be more investigated, defining a general multi-scenario approach. Besides, 
the introduction of photogrammetry into the process as a tool to acquire an additional 
level of detail may be of particular interest. Scan-to-BIM and reality-based modelling from 
classified data may be specific topics to analyses, supporting the point cloud seg—mentation 
purposes.A final goal concerns the creation of a classification framework that is more user— 
friendly for non-experts in the field, broadening its application to different disciplinary areas. 


Notes 


[I] The presented research is the result of the joint work of five authors. M.R. took care of the Introduction and Conclusions, 
E.G. prepared the State of the art, the methodological workflow and run the case studies, S.T. supported the methodological 
workflow and experiments, FF. and FR. supervised the work and reviewed the paper All authors shared the analysis of 
experiments and results. 
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Abstract 


In this work we show how Physically Based Rendering (PBR) tools can be used to extend the training 
image datasets of Machine Learning (ML) algorithms for the recognition of built heritage. In the field 
of heritage valorization, the combination of Artificial Intelligence (Al) and Augmented Reality (AR) 
has allowed to recognize built heritage elements with mobile devices, anchoring digital products to 
the physical environment in real time, thus making the access to information related to real space 
more intuitive and effective. However, the availability of training data required for these systems is 
extremely limited and a large-scale image dataset is required to achieve accurate results in image 
recognition. Manually collecting and annotating images can be very resource and time-consuming. In 
this contribution we explore the use of PBR tools as a viable alternative to supplement an otherwise 
inadequate dataset. 


Keywords 
synthetic dataset, image recognition, visual programming language, physically based render 


Ais fitias NTE RAE 


Silla Haili tii : 
AAA MALE AS J 144 Alias 
fiit. INES (atte Aiea sana 
F we 71 TEET a 4 
(aera ALLI ser Lak 
alefrrrtisatasasth fh rrisaba il 4 alter" ai 
Dati ini ta TI Leds EAS ado ifbtiiiidtatla 
Asia. FRANE A. 
N oe Tee ae eee FIRST] 
LisfnffAnkLgoktotfh 5 iti nfalatitoc iL 140911 
IE EEUFNTETUPKISBRHENTR 
FSFETT ALI 


fii AE AA 
A i 


doi.org/10.3280/0a-686.34 


Fig. |. Conceptual 
scheme of the survey 
and construction phases 
of the textured mesh 
model. 


Introduction 


In the context of built heritage enhancement, the use of mobile computing technologies for 
its fruition can revolutionize the user experience [Barsanti et al. 2018; Lo Turco et al. 2019]. 
AR systems, if properly combined with ML algorithms, can expand the level of knowledge 
that can be accessed while observing the asset [Spallone et al. 2020]. While it is very easy 
to imagine the database containing the information associated with an architectural asset, it 
is less immediate to imagine the query needed to access and explore it. Considering that 
within the same database very different information is stored: referring to the history, the 
construction technique, the history of the architect, etc.; one can understand how solu- 
tions such as audio tours, information panels, or QR codes are not suitable to answer the 
subjective curiosity of the user [Andrianaivo et al. 2019]. With the help of a mobile device, 
starting from the recognition of the object itself, one could connect and reorganize all this 
information according to the user's preferences [Vayanou et al. 2014]. 

To enable this kind of navigation, one of the first steps is to ensure that the mobile device 
can recognize the object in the frame. However, while some disciplines already apply Deep 
Learning for image recognition [Norouzzadeh et al. 2018; Liu et al. 2020], research is not as 
flourishing for architectural feature recognition. 

This research work proposes a methodology to enrich the training dataset needed to build 
a software capable of recognizing the built heritage from pictures coming from a mobile 
device with the help of PBRT tools. Once the architectural artifact has been surveyed, its 
digital twin can be inserted into a modeling environment and used for the creation of pos- 
sible views, even improbable ones, expeditiously, taking into account different lighting and 
meteorological settings which could affect the picture taken from the end user 


Case Study Definition and Data Acquisition 


Given the still preliminary stage of the work, the Saracen Tower of Spotorno was chosen as 
a case study by the research team because of its small size and its position visible from differ- 
ent points of the town. For this specific use case, a three—dimensional model has been useful 
for the creation of photorealistic rendering used to train a ML algorithm. For this reason, and 
to optimize working time, it was decided to generate a photogrammetric model by carrying 
out a free-net adjustment with a subsequent assignment of the model scale by applying 
the method of least squares over 3 known distances, measured using a metric token (fig. |). 
The approach of using elements of known length is a cheap, expeditious and well-estab- 
lished procedure, both in the orientation of the photogrammetric block in industrial appli- 
cations [Luhmann et al. 2010], and in the survey of archaeological heritage [Nocerino et al. 
2013]. The important thing is to correctly size the supports taken as reference — they must 
be proportionate to the object to be surveyed —, the distance of the images, and the degree 
of precision and accuracy required by the final model. 

The aerial photogrammetric shots were taken using a DJI Mavic Mini drone, equipped with 
a 1/2.3” CMOS sensor. The dataset was integrated with some images taken from the ground 
with a Sony Alpha 6000 camera equipped with a 23.5x 15.6 mm sensor 


Fig, 2. Selection 
throughout the year 
of 30 different lighting 
conditions and relative 
camera positions. 


Despite the non-professional tools, it is now known within the scientific community that the 
processes of generating point clouds from georeferenced photogrammetric blocks provide 
excellent results even when the starting data is not a set of images acquired with a calibrat- 
ed photogrammetric camera. [Cardenal et al. 2004] 

The Agisoft Metashape software was used for the 3D model generation operations. 

The method used has already been tested and considered appropriate for the survey of 
elements located in the vicinity of the supports used as references but, despite the cost-ef- 
fectiveness of the process, even at greater distances the uncertainty is within a few centime- 
ters. [Calantropio et al. 2018] 


Dataset Construction Workflow 


As previously mentioned, the generation of the 3D model, starting from aerial and ground 
photos, was carried out using the Agisoft Metashape software. Since the final aim was to 
produce photorealistic renderings in a fast way, it was fundamental to optimize the mesh, 
firstly, to reduce the calculation time, secondly, to achieve a representation without gaps, 
without visible polygon edges and with a more homogeneous appearance.A dense cloud of 
5,198,304 points was used as input for the mesh calculation, resulting in a mesh of 345,280 
faces. Then, to decrease the complexity of the geometry, the coplanar faces were merged 
and the areas surrounding the tower were decimated. The output of this process was a 
mesh of 219,269 faces. 

An algorithm was written within Rhino's Visual Programming Language (VPL) environment 
to automatically generate the useful photorealistic views from this last textured mesh. 
Grasshopper was chosen as the programming environment because of its ability to natural- 
ly manage complex geometries. Moreover, being integrated within Rhino, it was possible to 
easily connect it to different PBR rendering engines (Rhino Render and V-Ray). 

In the algorithm, three working phases can be identified: 

. The identification of useful views around the case study. A hypothetical circular path was 
drawn around the tower Along this path 26 chambers were positioned, | 3 at one elevation 
and |3 at a slightly higher elevation, with the centre of the tower as the intake point. 

2. Through the analysis of the epw weather file of the location, the solar path was imported. 
The months between the summer solstice and the winter solstice were selected, and for 
each month 5 moments in the day were selected in order to have a render for each possi- 
ble significant position of the sun. 

3. The last step was to automate the rendering procedure of all the views for each chosen 
moment during the selected months. We have 26 chambers, for each of them 5 positions of 
the sun were selected for the 6 chosen months for a total of 780 images (size of each image 
480*480 px). The images were exported in .png format with the contour of the architectur- 
al object. It was decided on a first hypothesis to include as little context as possible and to 
avoid representing the sky, in order to prevent the ML algorithm that would be trained on 
this dataset from identifying features on objects or landscape components (clouds, bushes 
or other) other than the tower 

A first dataset built in this way was generated in about 3 hours. Many of these steps could 
be further automated, thus reducing the required time. (fig. 2) 
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Fig. 3. From the left: 
Classification pipeline 
architecture; Two 
intermediate outputs of 
the pipeline stages. 


Test Classification Pipeline 


To test ifthe data produced by the aforementioned workflow is suitable for training a clas- 
sification algorithm, a naive image classification pipeline has been created. This has also been 
useful to identify possible problems with the generated images, such as the issue that will be 
mentioned in the next paragraph. 
The architecture of the pipeline is outlined in Figure 3.The first block is in charge of import- 
ing the data as it is. Since in this case we are using a simple binary classifier, we have four 


+ 


different types of data. The two types of data used for training are the pictures generated 
algorithmically and pictures coming from a public dataset of building facades for the other 
class. The other two types of pictures used for testing the classifier are real pictures of the 
tower taken with a camera and other pictures coming from the aforementioned public 
dataset. The pictures used to train the classifier are then elaborated in the 'Augmentation' 
and ‘Preprocessing’ blocks of the script to be ready to be used from the classifier The test 
pictures only have to be preprocessed. The classifier is an implementation of a Stochastic 
Gradient Descent (SGD) [Bottou Leon 2010] classifier which, even if not tuned, allows us 


to draw some conclusions about whether the data we generated can be useful for image 
recognition purposes. 


Original (BW) HOG 


Dataset Construction Improvements 


After training and testing the classifier on the first dataset, an overfitting problem became 
immediately apparent. The images, as previously described, were taken with the same 
camera settings, the same perspective and the tower was always in the center of the 
picture. As can be seen in Figure 3, to allow the images to be processed by the classifier, 
they undergo several levels of pre-processing, including the removal of color information. 
They are then processed by a ML algorithm that calculates the Histogram of Oriented 
Gradients (HOG), a technique used to select the most interesting features within the 
image from a software perspective. 
f the automatically generated images have a low degree of variability in some regions, 
which is visible in the HOG data, there is a risk that the classification algorithm will learn to 
correctly classify only those images with this degree of variability, ie. the synthetic ones, and 
misclassify the real images. 
To solve this problem, some degree of variability in the relative position of the tower in the 
image had to be included. The initial VPL algorithm has been modified, spheres have been 
constructed on the point on which the cameras were initially positioned and for each image 
the camera has been positioned on a random point belonging to this sphere.A similar solu- 
tion has also been used for the camera target, with this stratagem the images no longer have 
the same coordinates and our case study is never in the same position within the image. 
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Conclusions 


Even though it has not been tuned, the classifier scored a~ 70% of accuracy in distinguishing 
images of the tower from images of other buildings. 

The benefits of the presented approach versus a more traditional ground survey are clear. 
In a comparable time we can obtain much more data about the case study allowing to 
achieve promising results in image recognition with ML algorithms. 

With the tested classification algorithm, the background of the rendered 3D object plays 
an important role as it is processed along with the subject during the pre-processing phase 
and the resulting information is used during the training of the SGD classifier The output of 
the render should in fact have in the background different images which are similar to the 
true background of the real building, thus adding ‘noise’ behind the subject and reducing the 
risk of ‘overfitting’ the classification model, which would decrease its accuracy. 

Including the surrounding scenery into the rendering therefore means providing a more 
complete context to the subject of the survey. This can be achieved by taking some addi- 
tional pictures with the drone or by using inexpensive hardware to collect spherical images. 
This information can be augmented using a 3D rendering engine to include night or dusk 
settings as well. Moreover this technique also provides the operator a mesh with is useful in 
the mentioned AR applications. 
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Abstract 


The Diocesan Museum of Amatrice, razed to the ground by the 2016 earthquake, exhibited works of 
sacred art but also materials on the building, the former church of Sant'Emidio, and on the territory. 
The grave loss provokes a reflection on the psychological significance of heritage, the lack of which 
creates disorientation among the population. Technologies help to regenerate the bond with the ter- 
ritory, to search for one’s own identity not only in the tangible dimension, which is hardly visible today, 
but also in the relationship between objects, stories and meanings to be reconstructed. The research 
proposes a path of re-appropriation that identifies the semantic domains of the community and 
associates them in the territory through the dissemination of traces: images and stories are the new 
points of contact georeferenced around the voice of the protagonists. Two multimedia installations 
emphasize the evocative interaction between reality, memory and reproduction of the intangible to 
promote the common good of collective memory. 


Keywords 
cultural heritage, identity, storytelling, living experience, immersive technology. 


STORYTELLING 


PLACES PEOPLE 


223 doi.org/10.3280/0a-686.35 


Human Intelligence for Sensitive Places 


The research project intends to represent part of the lost heritage through an immersive 
and interactive system aimed at the experiential involvement and dissemination of the con- 
tents of the Diocesan Museum of Amatrice with visual storytelling and interactive iconogra- 
phy methods. The museum before its physical reconstruction can be expanded through an 
installation of the Digital Living Library, a solution to make the oral history of the territory 
alive through the direct testimonies of citizens. According to the Nobel Prize for Economics 
Amartya Sen, social participation depends on what he defines as capability, that is, the ways 
that all citizens need to be able to exercise their rights as a practical expression of freedom 
[Saito 2003]. The research goal is to investigate a new format of experience, supported by 
new technologies through the application of two fundamental principles: emotional story- 
telling, which addresses the need of citizens to connect emotionally to places before getting 
involved in a cultural experience; and the suspension of disbelief, which highlights how peo- 
ple tend to approach this form of interaction mainly for reasons of leisure and socialization, 
so learning becomes a consequence of involvement without making the didactic purpose 
explicit. It is a process that supports the development and enhancement of specific but also 
general “skills” that can allow access to knowledge. The exercise of memory develops a cog- 
nitive sense that makes us reflect on the material aspects and the psychological state, in the 
cyclical “return” to the places that were and in the rediscovery of the very meaning of living. 
It is important to underline how sharing the different expressions of intangible heritage, as 
a participatory cultural activity, is central instead of somewhat peripheral in intervention 
in earthquake-hit territories. Instead of adding a level of subjectivity to tangible forms of 
heritage — buildings or collections —, these being lost or heavily damaged following seismic 
events, the multiplicity and diversity of the inhabitants’ stories, memories, and experiences 
become the only possible way to preserve what remains in memory. 
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Fig. 1.A mobile 
interactive booth for 
installed in the civic space 
of Amatrice. 
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Fig. 2. Algorithmic 
generative 
representations for the 
reconstruction of the 
Amatrice. 


Moreover, this seems significant when time and money constraints often dictate that the 
cultural loss resulting from an earthquake is, at best, the last to be healed. If putting the 
rubble back together is not always possible, the use of technological but human—centered 
solutions can instead help to put together the experience and widespread knowledge of an 
entire community. Representing culture and human heritage in the digital age is the contem- 
porary challenge for designers figuring out interiors for community centers, like a museum 
or a library or the new hybrid spaces designed as urban interiors [Amoruso 2019]. 


RECONSTRUCT THE MEMORY 


Social Objects and Emotional Storytelling as Design Strategy 


Taking inspiration from the ‘human library’ phenomenon, that is, that a person's personal 
and daily history can become “culture” for other people. The experiential format for Ama- 
trice is related to culture as a common good produced “from below” in a way. The project 
starts from the principles of Design for All [Stephanidis et al. 1998] and Audience Develop- 
ment [Bollo et al. 2017], intended as levers to expand cultural participation. Specifically, a 
Design for All approach ensures the physical and cognitive accessibility of the environments 
and tools with which visitors interface. This concept has significantly developed in the Hu- 
man-Computer Interaction (HCI) literature following numerous researches [Stephanidis 
et al. 1998]. In terms of Audience Development, the project aims to reconcile the possibility, 
for people accustomed to enjoying cultural experiences, to have a ‘deep’ heritage experi- 
ence, ‘deepening’ the so-called ‘diversifying’ The project proposes two key concepts: the 
‘social object, a concept theorized by sociologist Jyri Engeström in a broader theory called 
‘object-centered sociality’ [Engeström 2005] and adopted in the museum by Nina Simon 
[Simon 2010]. “Third entities’ create processes of conversation and socialization between 
people who do not know each other Recognizing the importance of artifacts, Simon sug- 
gests rethinking the objects in a museum's collection as points of contact for conversation, 
with oneself, the cultural organization and its staff, and the other participants, present and 
future. It is essential to specify that this process gives technology a strategic role, enabling 
this ‘multilevel conversation’, both in synchronous and asynchronous mode. In the field of 
Audience Development, the research body Morton Smyth had already spoken of the need 
for cultural organizations to use ‘banner products, or artifacts “that speak loud and clear to 
novices, suspicious, apathetic to the frightened: this is for the likes of you” [Morton Smyth 
2004, p. 33]. Compared to the four categories of social objects identified by Simon, ‘Personal, 
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Active, Provocative, Relational’, the storyteller-booth falls within that of ‘relational’ objects, 
so defined because they explicitly refer to interpersonal use. As an emblem of a relational 


social object, Simon indicates the te 


ephone whose use presupposes the presence of an 


interlocutor ‘on the other side’, despite its invisibility. The phone connects us, despite the 
distance. It allows us to listen and, at the same time, allows the exchange of the voice. In 
short, it places us in a dialogic state of openness to others. The second design principle is 
‘suspension of disbelief, introduced by Samuel Taylor Coleridge in 1817. Typical of theatrical 


performances but increasingly 


+ 


adopted in the exhibition/museum environment (typically 
through immersive installations). The audience experiences the state of suspension of disbe- 
lief through extreme care and coherence of the contextual/scenic elements of the narrated 
world. Once this state is in the air; the viewer/visitor lives the experience represented as 
actual. This form of total absorption brings learning to an irrational level: the viewer lives the 
experience and only consequently finds himself having learned things. This dynamic is also 
the basis of gamification, which applies the technique of suspending disbelief in making the 
user act in the role of a character 


Technology—Driven Installations for Memory as Commons 


+ 


In the case of Amatrice, the triggering of the suspension of disbelief is fundamental.A mobile 
interactive booth and an immersive space were presented to the Amatrice administration 
and the World Monument Fund and received positive feedback. Monuments Fund (WMF) 
is a private non-profit organization founded in New York in 1965, created to preserve 


historical architectura 


artifacts and sites of historical and cultural significance worldwide 


through fieldwork, promotion, and training funds of local experts. The WMF has turned its 


attention to the Cola Filotesio Museum not only because it represents 


+ 


the cultural wealth 


of Amatrice but also about the history and identity bond that the population has maintained 
and maintains with the city and its territory. 


Among the social objects outlined by Simon, the project chooses relationa 


ones, that is, which 


explicitly refer to an interpersonal use. The immersive installation enables suspension of disbe- 


lief that is the imaginative condition in which the 


+ 


individual arises to escape reality and ‘enter’ 


the fiction. This concept, central to the performing arts, also asserts itself in the exhibition/ 


museum context, typically through immersive inst 
it is essential to translate knowledge into accessi 
and virtual nature, physical and digital, through informati 
dia stations dedicated to the exploration of digita 
environments placed in the museum rooms 


+ 


+ 


allations. In a society of continuous learning, 
bility. Furthermore, possibility between real 
on systems accessible 
collections, touch walls to form immersive 
to strengthen the communication of the rooms. 


ocally: multime- 


The installation of an immersive space with three interactive walls of 6 square meters, each 
with e-REAL technology by Logosnet, is planned according to p 


participation strategy. The user 


interfaces, with touch 


eye-tracking and gesture systems, stimulate multisensory perce 


of participatory scenarios inspired by the ter 
tive walls enhance the museum experience 


u 


blic communication and civic 
modes, but possibly expandable with 
ptions through the simulation 
ritory and the muse 
by making masterpieces of art accessible engag- 


ections. The interac- 


ingly and interactively. The system integrates the visualization of the works with infographics. It 


allows exploring a cultural microcosm with the touch of hands and the fusi 
D scenario where t 
experience the world from different points of view at the same time. 


al worlds. People are visiting a 3 


+ 


+ + 
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hey can inte 
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t glasses or wearab 
tively Javascript and Python, are the programming languages used to deve 
of the scenarios. A further objective is developing a mobile interactive 
ect direct testimonies from citizens on the heritage of Amatrice, life st 
city to be georeferenced. The booth allows the recording of vocal contributions, according to 
a series of topics, issues, and keywords divided into thematic sections: terri 


on of real and virtu- 


ract with natural gestures and 
The dynamic contents 
to be explored in an immersive way are usually developed within a virtua 
a graphics engine that allows 2.5-3D views withou 


Unity environment, 


e devices. C #, alterna- 


op the components 


booth intended to col- 


es, memories of the 


+ 


tory, history, places, 


traditions, people. Inside the mobile interactive booth, among the features provided, there is 


an interactive electronic whiteboard called e-Wall (writing, drawing, and painting, manipula- 
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tion of images and virtual objects). The system includes a voice recognition system that allows 
voice commands for the functioning of the e-REAL system, for example, through the use of 
“formulas” to make digital objects appear: 


Conclusions 


Therefore, the research project proposes a new idea of a temporary and experiential museum 
as a cultural garrison and for the transmission of memory and spatiality that digitally expands into 
the immaterial dimension of informal accessibility and discovery. Technologies are changing the 
relationship between users and the environment of use of cultural content in museums, libraries, 
and learning places. The environments are transformed considering their virtual extension and 
allowing a range of customization linked to content selection. To these, Manovich also adds the 
interactive cultural experience, the possibility of enjoying cultural experiences and products by 
visitors, textual, voice, and visual communication, and participation in a sort of information that 
regenerates ‘ecologically knowledge and its dissemination. Rebuilding Amatrice is a complex ar- 
tifice with an uncertain outcome. In respect of the cultural heritage, it finds its form through the 


values of living and the experience of those places that only its community knows. 
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Abstract 

From the cafés in the dreamlike square in Rimini depicted by Fellini in Amarcord to the movie theater 
in the village of Giancaldo depicted by Tornatore in Nuovo Cinema Paradiso, historic shops represent 
a rich and significant cultural heritage. The protection of historic shops is often the subject of specific 
dedicated regulations throughout the country, aimed at preserving their compositional identity and 
preventing the impoverishment of their urban image. In this context, starting from the analysis of 
future scenarios in the field of retail design, the proposed contribution aims to open new perspec- 
tives on the potential offered by the use of AR and Al technologies in the identity enhancement of 
the architectural heritage of historic shops. The idea, presented through a real case study, is that the 
concept of Augmented Retail can represent a strategy to achieve an Augmented Identity, changing 
the retail space from a ‘product window’ to an ‘experiential theatre’, 


Keywords 
AR-AI, historic shops, retail, heritage, valorisation. 
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The Shop as an Identity Value 


Historic shops represent a cultural heritage with a strong identity value, and this can be seen in 
the many artistic and cultural expressions in which the image of Italy is portrayed through that of 
its historic places: interiors, shop windows, overlooking squares and open spaces, urban views, etc., 
which give vividness with intense colours to the glittering image of the ‘Belpaese’. In this regard, 
the dreamlike narration of the city of Rimini by Federico Fellini in Amarcord is emblematic, as is 
the story of the movie theater proposed by Giuseppe Tomatore in Nuovo Cinema Paradiso. The 
ife of the community is marked by the interior and exterior spaces, by the scenery of the square 
characterised by the iconographic apparatus of neon signs) and the shops. The identity traits of 
the shops mark the succession of times and tastes: “it is the shop which best indicates the level 
of industrial and economic development of a country and which also best reflects the state of its 
culture and art” (Sabatou 1938/1984, p. 6, translation by the authors). At the same time, the shop 
represents the constitutive element of the urban landscape; as Guido Canella affirms, “the ‘living 
pictures’ of the shops facing and lined up on the street are breaks of a hoped-for city landscape, 
where [...] objective, symbolic, unifying, deforming intentions, corresponding to an aesthetic feeling, 
enter each time” [Canella | 984, p.2, translation by the authors] The historic shop substantiates the 
very image of the country, determining its perception from the outside and becoming fully part of 
the collective memory:"If Italy loses its shops, we lose Italy as we know it” [Petrini 2020, translation 
by the authors]. In the regulatory background, the Codice dei Beni Culturali e del Paesaggio (D.L. 
n. 42 of 22 January 2004) is not specific about this type of property and generically includes it 
among the “immovable and movable property of artistic, historical [...] and particularly important 
interest. The first explicit reference to “historical traditional premises’ was made in D.L. 91/2013: 
it is often the individual regions that enact specific provisions for the protection and enhancement 
of these assets in the form of regional laws. In recent years there have been many initiatives aimed 
at raising awareness of the issue and comparing experiences at national level. Many contributions 
confirm the attention paid by the scientific community to commercial establishments; these are 
almost always census activities, generally based on the architectural survey and cataloguing of 
historic shops and aimed at enhancing them in various ways (dedicated tourist circuits, restoration 
work, protection actions, also in terms of safeguarding their use, etc.). 
n this context it is possible to list some significant examples: starting with the cataloguing of 
Turin's historic shops, begun in 1985 [Tagliasacchi 1985], then systematised [Ronchetta 2001] 
and extended to the entire Piedmont region [Ronchetta 2008]. The cataloguing work, shared 
publicly through the MuseoTorino portal, makes it possible to consult documentation sheets 
and digitised materials on the individual businesses recorded, also by locating them on a map. 
Other examples are represented by the survey and cataloguing of historic shops in Lucca 
[Pellegrini 2001], aimed at documenting the technical expertise of the artisans and promot- 
ing restoration and enhancement actions, the cataloguing of commercial activities in Tuscany 
[Preite 2007], the census of historic premises in Friuli Venezia Giulia [Regione Autonoma 
Friuli Venezia Giulia 2010], by the proposal for a Multimedia Information System of the com- 
mercial activities in Bologna [Bartolomei 2013], by the census of historical premises in the 
Marche [Regione Marche 2014], which led to a dedicated guide, and finally by the census 
and photographic guide of the historical places of commerce and catering in the Veneto, 
from which the app Veneto su misura was produced. 

Among these initiatives is the activity of the research team of the Department of Civil and Envi- 
ronmental Engineering of the University of Perugia. With regard to the case study of the historic 
centre of Spoleto, in the context of the initiative Consulto su Spoleto (2017), starting from the 
analysis and highlighting of the critical points of the Building and Town Planning Regulations, the 
aim is both to draw up a census and typological analysis of the commercial activities present along 
the streets of the historic centre, and to identify a series of criteria to guide planning choices, also 
through the promotion of city branding actions. The question as to which instruments today al- 
low for the protection and, above all, the enhancement of historic commercial activities has not 
yet found an unequivocal answer; in fact, the debate, at times polemical [Adinolfi 2016], on the 
opportunity of fluid reallocation of protected commercial spaces t 


to new brands, even related to 
product categories very distant from the original destination, is still open. A controversial case in 
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this regard is the Santa Maria Novella railway station in Florence, a protected work by Giovanni 
Michelucci, where the relocation of commercial activities has meant that a brand of underwear 
has found a home in the former offices of a bank and a supermarket has moved into the space 
originally dedicated to worship.Although this stratification may create an interesting ambiguity in 
some respects (in this case, the overlapping of signs does not seem to be a reason for disorien- 
tation, but rather highlights the vitality of a place able to update and renew itself over time), the 
problematic knot of the ephemeral margin between protection and reuse remains open. 


The Shop as an Experiential Theatre 


In light of these considerations on the identity character of the historical shop, with the aim 
of tts enhancement in contemporary terms, it seems necessary to question the meaning it 
currently assumes. In other words, it is necessary to acquire a new awareness of the concept 
of ‘shop’, a concept that has undergone deep changes over time and is constantly evolving. In 
this context, tt assumes nodal importance to investigate the opportunity offered by the use 
of New Technologies, with specific reference to AR and Al, also as an evolved strategy for the 
protection of the material characteristics of places, given the current conception of the physi- 
cal retail space [van Escha et al. 2019]. Increasingly, the shop is not only a material place to buy 
products, but it becomes the ‘emotional centre’ through which communication between the 
brand and its target audience takes place [ Taylor, Zavoleas 2018].With the spread of e-com- 
merce, in fact, it is possible to buy anywhere and with just a few clicks: the relationship between 
buyer and product is mediated by a device and, in this sense, requires that the customers 
involvement takes place at other levels and through different channels. The physical shop, on 
the other hand, has to offer a different experience to e-commerce, possibly personalised 
and user-centred, capable of including those elements of memorability and engagement that 
determine both diversification from computerised purchasing and an effective support to it. 
For example, in today's advertising policy the flagship store represents an investment whose 
profit is estimated in terms of advertising rather than on-site sales to the consumer The store 
becomes almost an advertising space, similar to that available in publishing, television or other 
media. The objective underlying this dynamic is the evolution of the consumers role into that 
of ‘consumer—actor’, as the user; having identified with the brand and recognised its values, 
becomes its ambassador and testimonial. The physical shop, both through the architecture of 
the place, the design of the interiors and of the retail space, and through communication ac- 
tions and an increasingly advanced use of technology, becomes the scene of a user experience 
aimed at the progressive discovery of the brand through strategies of engagement, customi- 
zation and strengthening of the community [Riewoldt 2002, Marchetti 2009]. The objective 
of engagement is to involve the user in an immersive experience, whose location par excel- 
lence is the physical shop: dedicated entrance, shop window and room/corner are the nodal 
points of the connection and emotional exchange between user and brand. An emblematic 
example, which embodies this branding strategy through AR applications, is represented by 
the Nike flagship store in NY, where there is a ‘Running Trial Zone’ that allows visitors to test 
the most suitable shoes for jogging and competitive running on a treadmill facing a wall/screen 
that simulates the immersion of the runner in Central Park. The aim of customisation is to 
offer the user a personalised and customisable experience; the shop respects and satisfies the 
diversified needs of a multiplicity of customers, proposing distinct paths according to the type 
of user There are numerous examples of this strategy: from the Fragrance Lab (multi-sensory 
interactive areas for personalised fragrance exploration and product customisation) in one of 
Selfridge’s London locations to the Hunter pop-up (an interactive and immersive greenhouse 
designed as a test room to simulate the use of clothing) installed in NY Central Terminal. Finally, 
the aim of the community is to use the physical location of the shop to build user involvement 
within a 24/7 connected community through events, gamification, contacts and sharing. The 
architecture and scenery of the store become a source of wonder and an object of sharing; 
digital communication makes the spread of the brand, its values and therefore, its products 
viral through the users themselves. In this sense, the example of the Samsung 837 store NY 
is emblematic: a multifunctional store, which incorporates, in addition to its usual function as a 
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shop, those of a theatre, cafeteria, museum, games room and, last but not least, a multimedia 
tunnel for an immersive experience, whose external transmission is entirely entrusted to the 
sharing of selfies by users/ambassadors on their social profiles [Galasso n.d.]. 


The Shop From Identity Value to Experiential 
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Fig. |. Barbieria Lolli in 
Perugia, the interior of 
the historic shop (left) 
and the project of the 
new interactive shop 
window (right). 


the showcase Is conceived as a virtual surface subject to continuous changes thanks to the interac- 
tion with passers-by, offering illusionistic experiences in the space of everyday city life (fig, 1). The 
new shop window is set up on the basis of factors that do not strictly concern the enhancement 
of the material goods on offer, but as a device for activating the identity value of the University of 
Perugia (conveyed by the new logo) while protecting the historical character of the place. 
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Abstract 


The research project aims to acquire data on the impact of an environment on humans. The pro- 
cess is based on the use of the EEG helmet synchronised to GPS: the EEG helmet is a device that 
non-invasively records 14 channels of the human brain through electrodes and, through an algorithm, 
transforms them into cognitive states in real time. Starting from this raw data, through the ‘circumplex 
model’, cognitive states can be transformed into emotions with the aim of showing what people on 
average feel in a given urban space. The data, which are linked through GPS to a position in space, 
are empirically recorded with statistically significant samples from which emotions are reprojected 
onto the mesh of the digitally reconstructed environment through a process of photomodelling. 
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In addition to the EEG data,the geographical position is acquired in real time at a regular interval 
of one second; thanks to the timestamp, it was possible to synchronize the GPS and EEG data. 
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The experimental route leads to the collection of data on the position and cognitive states 
of several testers. It was therefore necessary to find a way to compare the emotions of dif- 
ferent testers according to their spatial position. By means of an algorithm developed in the 
visual script language Grasshopper for the Nurbs environment of Rhinoceros, it is possible 
to import CSV data of the experiments containing synchronised coordinates and EEG data 
and identify at which positions the observations were taken from contiguous and thus com- 
parable points. The developed algorithm [Bianconi et al. 2019] [Bianconi et al. 2020] then 
identifies the comparable recordings and calculates the average of the EEG and GPS values 
belonging to the same cell; in this way the average of the observers in space is obtained. 
The representation of the data is possible using as a base a mesh model of the terri- 
tory (DEM) under analysis; the model is obtained through a process of photomodel- 
ling via google earth pro photos with 3d buildings enabled. An attribute is added to the 
mesh base at each vertex, which corresponds to an RGB vector representing the aver- 
age value in the circumscribed model of the sensation felt by the observers. Finally, the 
faces of the mesh are coloured by interpolating the vertex colours in a linear fashion. 
The representation thus obtained has a double objective: to store the instrumental data and 
to represent them in a comprehensible way (figs. 2-3). 


Fig. |. The research area. 
The centre of Terni. 


The Case Study: Terni City Centre 


An illustrative case that shows the value of the analyses are the studies carried out in a 
portion of the historic centre of Terni that from Piazza Tacito, across the Corso, reaches 
Palazzo Spada (fig. I). The area under examination constitutes a section of considerable 
interest because within it there are many peculiarities. In an extremely circumscribed space 
there is a pedestrian area, intersections with roads, a square with a recent arrangement of 
street furniture and a square with an important Renaissance palace. And it is precisely the 
way in which these elements influence the emotional state of the user that is the subject of 
the analysis in question. 

From the data acquired and processed it emerges that the area can be divided into four 
zones according to the emotions aroused in the observers.At the starting point, on average, 
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Fig. 2. Processing obtained 
through the developed 
algorithm. Each colour 
represents an average 
emotion felt in space. 


a feeling of strong stress emerges due to the area highly congested with car traffic; continu- 
ing, the pedestrian area represents an area that oscillates between serenity and calmness 
except near the intersections with the driveways. In the final part of Corso Tacito a gradual 
transition from a state of calm to a high degree of stress in the vicinity of Palazzo Spada is 
clearly visible. This analysis shows that the pedestrian area of the city’s main street is per 
ceived as a comfortable space for users, but the fact that it is located between two highly 
trafficked squares limits its potential. 


Conclusions 


The proposal aims to define methodologies for data collection and interdisciplinary inter- 
pretative criteria to understand the impact of the environment on man, with the implicit 
objective of being able to direct design choices to build places that create wellbeing. 

With an interdisciplinary approach and through the integration of multiple devices for 
detecting perception, it is possible to analyse what people feel and understand how the 
environment implicitly influences their emotional state and which places and conditions 
promote wellbeing. 

Digital becomes a true Computer-aided Design tool, with digital data used not only to 
show the visible but also to make visible what is implicit in the relationship between man 
and the environment. The aim is thus to conform an innovative methodology that supports 
an operational verification of the meaning of places, which analytically focuses on emotions 
and meanings and can be used as a strategy for studying not only the territory and the 
environment, but also the landscape, understood as the result of the perception process. 
The creation of new data and their representation are the basis of new methodologies of 
investigation through the use of Al for the design and optimization of urban spaces. 
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Fig. 3. In the perspective 
section, one can see the 
division of the average 
emotions felt during the 
experiment. 


Fig. 4.Via the circomplex 
model is to represent 
through the colors the 
emotions. 
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Augmented Reality as a Research Tool, 
for the Knowledge and Enhancement 
of Cultural Heritage 
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Abstract 


Starting from a reflection on the new role of digital technology in this pandemic situation, and partic- 
ularly the role of AR for Cultural Heritage, this paper presents two AR applications. The first is used 
to share the analyses carried out on the Pantheon surveys realized between the seventeenth and 
nineteenth centuries. The second is about the illusionistic dome in the Church of Sant'Ignazio in Rome, 
realized by Andrea Pozzo. These two applications can display three—dimensional models superimposed 
onto images and original drawings, and also two-dimensional contents: the results of graphical analysis 
carried out on digital copies. The final aim of this ongoing research is to develop some AR multimedia 
content, linked to semantic concepts. 
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augmented reality, cultural heritage, geometric analysis, 3D survey. 
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Fig, |. Over the drawings, 


side menus show 
the main themes of 
comparison. 


Premise 


This last year, with the restrictions due to the Covid-19 pandemic, the growth of digital 
technologies, applied to activities related to cultural heritage but which cannot be accessed 
physically, have accelerated in an unprecedented manner, thus generating innumerable spawning 
of network applications, a kind of "digital golden rush" [Concas 2021, p. 15].A profound evalua- 
tion and reformulation of digital methodologies and technologies are thus deemed necessary, for 
they have led to a turning point, a kind of "destructive innovation" [Santamaria 2021]. 

Virtual, mixed, and augmented reality have all become common terms in recent years. How- 
ever, the relationship between real and virtual is not only of interest at present. Since the 
1990s, Tomas Maldonado initiated reflections on the theme of the relationship between the 
real and the virtual according to which a (real) painting prefigures a space of the (virtual) 
pictorial scene, in a relationship that transcends the boundaries between space and time 
[Maldonado 1995, p. 162]. In this context, some examples are the eighteenth-century views 
of the historic center of Dresden by Bernardo Bellotto, in which the space depicted in the 
(virtual) painting is used as a reference model for the post-war reconstructions (real) of 
the old city, destroyed in the 1945 allied bombings. The same space-time relationship is 
evidenced in a 2016 installation, where the view of Dresden from Elba's right bank depicted 
in a 1748 painting (virtual) by the Venetian painter is viewed within an empty frame (real) 
[Dal Pozzolo 2021, p. 145]. 

Augmented reality certainly represents the most appropriate digital tool to permit the ob- 
server to move within a hybrid space, that is physical and virtual. Already in 1995 [Milgram 
et al. 1995], Augment Reality (AR) was situated within a space defined as a Virtuality-Reality 
Continuum, and delimited by two extremes, the real and the virtual world. According to the 
definition by Nofal [Nofal et al. 2017], AR moves within a hybrid space (called Phygital, a 
fusing of the terms physical and digital), where the boundaries between physical and virtual 
space are not necessarily clear or defined, thus delineating various degrees of AR and varying 
uses of virtual contents. These can be described according to three typologies: 

|) the first type, multimedia, where information related to an object, physical and real, are 
connected through various media (texts, images, data); 

2) the second type, hybrid, to the real object, a two-dimensional print or drawing, are con- 
nected data and models, which may themselves be two or three dimensional; 

3) the third type, three-dimensional, to the real object, that is the physical space enveloping 
the observer data and virtual 3D objects are superimposed. This represents the most ad- 
vanced level in the use of AR. 
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Fig. 2. Desgodets' 
interpretation of 
roundness versus 3D 
survey: our comparison. 


Introduction to AR Applications 


The applications described in this article are at a stage prior to the final aims of the research, but pres- 
ent interesting results with potential developments yet to be explored. These applications integrate 
the usual display mode of three-dimensional models superimposed onto images and drawings with 
two-dimensional contents, that is to say the results of graphical analysis carried out on digital copies. 
The fundamental issues in these types of applications are related to the need to be able to 
identify a rational selection and synthesis of the scientific contents to be displayed which, though 
necessary, given the effectiveness and distinctness of the means of communication used, must 
not compromise the scientific result. The project [|], whose initial report is presented here, 
addresses these issues by identifying in each of the themes presented, the narrative lines of the 
contents, allowing different levels of detailed studies, through a sharing of information that is 
gradually more complex, to meet the diverse interests of distinct users [2]. 

From a technical point of view, the applications were developed on a platform that simplifies the 
AR experience project, is compatible with all kinds of devices [3],and guarantees sufficient accuracy 
in the overlap of various levels of information. Its versatility has allowed an ad hoc development, ad- 
dressing the categories of possible users and the sites where the AR experience may be employed. 
The two applications, described below in detail, refer to two different contexts but share the 
same objectives: to convey unusual items of knowledge, perhaps not easily understood imme- 
diately, but that aim to show how the drawing and the use of AR technologies can reveal the 
geometries underlying the apparent forms, and also bring them to the attention of the general 
public and stimulate the more inquisitive ones to research further in more appropriate locations. 


AR-Pantheon: Survey and Drawings 


The objective of the first application was to use AR to share the analyses carried out on the 
Pantheon [4] surveys realized between the seventeenth and nineteenth centuries, in particu- 
lar the studies carried out by the pensionnaires of the Académie des Beaux Arts in Paris and 
by the students of the Academy of San Luca in Rome. An analysis of the survey drawings 
tells us how, over time, the architects looked to antiquity, allowing them to consolidate their 
knowledge on the construction and conservation history of the monument. Consulting the 
original drawings in the archives of the aforementioned Academies [AABA] [Marconi 1974] 
made it possible to closely analyze the documents to identify the marks relating to the con- 
struction lines and the holes of the compass points. 

The 3D survey of the Pantheon, achieved through the use of integrated systems (photo- 
grammetry and laser scanner), made it possible to carry out an accurate comparison be- 
tween the archival drawings and drawings of their present state. 

The results of the analyses and of the superpositions were processed so they could be dis- 
played by the AR application (see note 3), which uses marker-based tracking technology 
[Bekele 2018, p. 5]. The application was realized with a dual purpose: on the one hand, the 
possible use in an exhibition of the drawings of the Pantheon, on the other, directly in the 
archive, to show the analyses carried out, superimposed over the original drawings. The appli- 
cation can also highlight the different interpretations of the geometry of the Pantheon and the 


+ 


transformations undergone in its transition from an ancient monument to a church. 
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Fig, 3. Left: Church of S. 
Ignazio, Rome, interior; 
right: a demo of the use 
of the AR application 
“AP_Andrea Pozzo_ 


Augmented Perspective”. 


When the drawings are displayed, AR side menus (fig. |) introduce the main items, such as 
the geometry and size of the Pantheon dome, to compare how the different authors of the 
various survey drawings interpreted them (fig. 2). 

Thanks to the use of geometric elements (the circumferences, axes, tangents, etc.) and 
three—dimensional models (the cylinder and the sphere) superimposed on the plans and 
sections, our augmented reality application summarizes — by relating the drawings in space, 
the plans (virtual) and the sections (real) — the different analyses of the monument that the 
architects of the past carried out. 


Andrea Pozzo Augmented Perspective 


The second application refers to the illusionistic dome in the Church of Sant'Ignazio in Rome, 
the work of Andrea Pozzo (1642-1709) [5]. 

The development of an AR application on this theme began with a consideration of the 
spatial location of the canvas depicting the illusionistic dome (fig. 3) horizontally situated 32 
meters above ground, at the intersection between the nave and the transept of the church 
of Sant'Ignazio in Rome. The canvas, simulating a real dome in perspective, is correctly visible 
from a preferential point of view indicated on the ground by a marble disc, which allows 
visitors to comprehend the perceptive artifice. The technique used (tempera), over time, 
has developed a blackening of the pictorial surface [6] so that one cannot appreciate it as it 
would have been in 1685, the year it was installed in the cross-vault. So an AR application 
was developed, with the intent to convey not just the wealth of details of the painting but 
above all the perspective scheme that allowed Pozzo to compose the illusionistic dome. 
Thanks to a simple setup of two display panels (fig. 3), the application allows an augmented 
interaction with a scale reproduction of the painting. 

The application is divided into three sections: users can choose the content according to 
their interests on the subject selecting different levels of information detail. 

In the section ‘artistic historical research’ one can compare different works of illusionistic 
domes ascribable to Pozzo, collated in the different phases of the historical—archival study. 
Thereby offering the possibility of partially superimposing over the marker, depicting the 
painting, the drawings and sketches of other illusionistic domes by Andrea Pozzo, compared 
instantaneously thanks to the virtual space of the AR. 

In the ‘surveys’ section, it is possible to view, aligned with the marker the high-resolution pho- 
to, to analyze the pictorial consistencies of the architectural elements of the work down to 
the smallest details, which are otherwise invisible when viewing the original, given the height 
at which the canvas is placed. 

The section ‘geometric analyses’ contains the graphic elaborations carried out on the painting, 
beginning with the study of the two methods that Andrea Pozzo describes in the two volumes of 


his treatise Perspectiva pictorum et architectorum (Vol. | 1693,Vol. || 1700). Several layers gradually 
are superimposed over the marker evidencing the perspective restitution operations carried out 
to retrace the architectural section from the perspective drawing. 

With the application, one can visualize a section of the model of the church and relate it to the 
plan (fig. in front page). The aim of the model is twofold, to represent the church with the virtual 
reconstruction of the dome designed by Pozzo and at the same time show the perspective meth- 
od underlying the same restitution by retracing, in reverse, Andrea Pozzo's modus expeditissimus. 


Conclusions 


These two case studies show how we have used AR to share the analyses carried out on 
drawings and canvas. In the future, this kind of application should integrate databases orga- 
nized according to semantic structures [Canciani 2020], which can connect related research 
fields. Our aim is to develop on-site AR applications, using markerless tracking, to introduce 
3D models as well as two-dimensional contents surrounding the users. 


Notes 


[1] nn the project is the research team of the Department of Architecture (Roma Tre) consisting of M. Canciani, G. 
Spadafora, M. Saccone, A. Camassa, in collaboration with the ENEA research team coordinated by M. Mongelli. 


N 


] It is understood that the level of detail proposed in the project is consistent with the methods designed for these two 
particular AR applications. 


[3] The commercial application used is blippAR®. 
[4] Selected were some of the results of the doctoral research: Disegnare il Pantheon. La pratica del rilievo nell'insegnamento 


accademico del XIX secolo e nel disegno digitale contemporaneo presented by Mauro Saccone in the XIX cycle of Architectural 
Doctorate: innovation and heritage (tutor Marco Canciani). 


O 


] The application presented here draws a some results of the doctoral research, currently being coi e congiug nere il 
finto col vero” — Geometria e architettura nella finta cupola di Andrea Pozzo a Roma, presented by Antonio Camassa, I cycle 
of the Architectural Doctorate: innovation and heritage (tutor Giovanna Spadafora). 


[6] As the early | 8th century, the illusionistic dome was not visible, as reported in [Montalto 1962]. 
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Augmenting Angri: Murals in AR for Urban 
Regeneration and Historical Memory 


Alessandra Pagliano 


Abstract 


Augmenting Angri is a project aiming at preserving the memory of about twenty murals located in 
the historical centre of the small city of Angri (SA) using ICT and in particular Augmented reality. 
The project has been carried on in 2020 with the students attending the integrated course of Visual 
Expression (professors A. Pagliano and P. Vitolo) in the Masters Degree Course in Design for the 
Built Environment. 

Augmenting Angri consists of an innovative valorisation path in Augmented reality which expand the 
narrative message of the old murales thanks to the digital contents (audio, video, animations and 3D 
models), sometime replacing them back, in case of totally disappeared murales. 


Keywords 
street art, augmented reality, cultural heritage, artivive, murales. 


Fig. |. Augmented digital 
contents for the mural of 
Vittorio Miranda, today 
disappeared, screenshot 
from the app. 


From Graffiti to Street Art in Contemporary Cities 


Street art emerged as an artistic practice since the 1960s in the United States, in form 
of graffiti, namely the tags of the writers who usually are young people from degraded 
suburbs who leave a testimony of their problems on empty walls, thereby performing 
an unauthorised and illegal action. Today, street art has evolved considerably, abandoning 
graffiti in favour of painted images, often characterised by a high artistic quality, which 
has attracted the interest of critics and art historians, becoming today one of the most 
established forms of urban art. 

Abandoned or degraded spaces are enhanced with fast, low-cost and high-impact inter- 
ventions that sometimes become the spark and the starting point for a following physical 
redevelopment and social regeneration. In the abandoned places, in the residual spaces 
[Bauman 2002], in those buildings without any quality and in the so called 'non—places' 
[Auge 1996], urban art is a powerful tool capable of reopening the dialogue with the urban 
context through imagination, transforming these inexact spaces into active elements of the 
urban and social landscape. The modification of an abandoned space through urban art 
intervenes on the cultural and perceptive aspects capable of influencing both the physical 
space and the inhabitants’ habits, superimposing new signs on the pre-existing ones and 
thus creating a ‘new space’ through the interaction of the physical volumes and the illusory 
appearance superimposed on them. So, the murals adopt those walls, transforming and of- 
ring them a new role in the surrounding space. The shape, size and subject of a mural are 
thus a spatial response to a latent urban question. 

When murals are socially accepted for their aesthetic contribution to the urban environ- 
ment, the city appropriates the works. In such circumstances, citizens strongly assert their 
need to own and preserve these artworks, requiring them to remain. The request for con- 
servation and permanence emerges a posteriori, thanks to a slow process of social recogni- 
tion of their value. 

But the problem related to the maintenance of murals is an ongoing issue, which goes 
from the recognition of the mural as an artwork to the need for its protection in order to 
preserve and pass on its value to future generations. The painter holds the paternity but 
not the ownership of the artwork, since it was painted it on other people's walls. There- 
fore, street artists are used to accept the temporary nature of the artwork itself, probably 
destined to be removed by the will of the real owner of the wall. Even in case of famous 
artists, transferring murals to museums is inappropriate because the very meaning of the 
mural would be altered if removed from the urban context which generated it, but also 
because it could be done against the will of the author, who instead generally accepts his/ 
her artwork to be perishable. 
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Fig. 2. Giving a new 
interactive depth to the 
painted image in the 
Ativive bridge app. 


Street art is always exposed to the ever-changing conditions of the street and It's quite 
impossible to protect murals from the rain, snow, direct sunlight, or intense cold; protecting 
a street artwork, as in the very debated case of Banksy's mural in Via dei Tribunali (Na), 
recently covered with a glass that protects the painted image from the weather, profoundly 
alters the inner nature of a street mural. 

This suggests that street art cannot be assumed as “finished” artwork, strictly linked only to 
the author, because the constant exposure to change is its inner nature and, perhaps, the 
only real way for it to remain authentic. 


Augmenting Angri: Augmented Reality to Valorize the Murals 


In terms of street art's protection and heritagization, the lack of defined and consolidated 
guidelines, due to both the high variability of the materials used by artists and the issues 
related to the ownership of works, produces an immobility that is far more damaging than 
the weathering itself. 

Augmenting Angri is a project aiming at preserving the memory of about twenty murals 
located in the historical centre of the small city of Angri. The project has been carried on 
with the collaboration of the students attending the integrated course of Visual Expression 
(professors A. Pagliano and P Vitolo) of the Masters Degree Course in Design for the Built 
Environment. The area of the old Angevin village is formed by two orthogonal streets that 
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shap four insulae. The road axes are named according to their orientation with respect 
to the cardinal points (Via di Mezzo Sud, Via di Mezzo Nord ...) and extend close to the 
medieval castle. The murals in the Vie di mezzo were painted between 1982 and 1983 and 
constitute a pioneering urban art project, to bring collective attention back to the cultural 
value of Angri's historic centre, which was suffering from serious deterioration and subse- 
quent abandonment after the 1980 earthquake. 

The initiative was promoted by the municipal administration. Twenty artists, led by Gianni 
Rossi, transformed the four streets into an open-air gallery, by painting murals of various 
sizes and different subjects, which were inserted on the facades according to a variety of 
ways, in order to suit the morphology of the walls, with windows and doors instead of 
treating the facade as a blank canvas. The artists acted as spokesmen for the collective and 
personal malaise of the post-earthquake years, or in other cases they represented the most 
characteristic elements of Angrian identity, such as the agricultural vocation, deep religious 
devotion or local historical episodes. 
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Severely degraded, very little remains today of the original murals, but a good photo- 
graphic documentation and the potential of digital technology have enabled us to re- 
propose each mural in the original locations, bringing them to life by means of short 
animations. In some cases, they totally disappeared, completely erased by the renovation 
of the building facades: in fact, some owners took over the buildings when the mural was 


at an advanced stage of deterio 


ration and they were unable to perceive original artistic 


value and the role that the mural had for the collective memory. They simply identified 
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them as signs of decay and removed them. 
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tion of the still remaining painted images, and since we also did not consider appropriate 
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tended as a reproduction, a digital copy, even if faithful and accurate, but as the 
(both digital and Physical) that provides further knowledge. 
Ben Hicks defines this new digital asset as ‘digital twin’, remarking the differences with a 
tal prototype or model “as an appropriately synchronized body of useful infor 
mation (structure, function, and behaviour) of a physical entity in virtual space, with flows of 
information that enable convergence between the physical and virtual states’ [Hicks 2019]. 


According to the previous definition, digital twins in not a mere copy of the real object but 


a new object with data, functions, and features of the real one but 
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added by the communi- 


cation capabilities of the digital world. The reconfiguration of the cognitive process, typical 
of digital technologies and in particular of augmented reality, takes place only when the 
digital contents are experienced and lived, ¡.e. “activated”, by an observer. In fact, the designed 
semantic link is enabled and implemented only through individual interaction and experi- 
ence with the designed knowledge contents. Today, the ability of cultural heritage to arouse 


emotions, establish links and stimulate curiosity can become effective if we take int 
that contemporary users have completely changed: they are bo 
‘digital natives’ with different expectations, pre-existing knowledge, historical/cultural back- 
grounds and interpretative strategies. The students of the 
course were encouraged to address thei 
evocative power of each mural to develop 


is, in fact, a good 


binder to combine emoti 


Design for the Built Environment 
r project to this new audience, uncovering the 
and enhance its storytelling. The narrative aspect 
ons with information. Murals’ painted scenes are 
the representation of a single moment, which remains eternally frozen and which the artist 


+ + 


o account 
th ‘digital immigrants’ and 
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subtracted from a long-envisioned sequence of gestures and positions, by painting a crys- 


tallised motion. 


gesture, a gust of wind, a fleeting glance, a 


Each image has been analysed for its ability to suspend time with a hinted 
facial expression that become permanent. In this 


way the time is allowed again to flow before and after that suspended portrayed moment. 


The brief animati 


+ 


ons, designed to give life 


to the static painted scene, are slight variations of 


the portrayed configuration, as a simple temporal expansion of that moment. Augmented 
reality allowed us to integrate or change the state of each mural without compromising the 


physical state. The link between AR and m 
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urals' visual and pictorial aspects required a deep 


disciplinary study in terms of representation as a possible product of new ways of looking at 
reality, new perceptions and therefore new ways of interacting with perceived space. 
Furthermore, the images of the Angrian village murals were also interpretated according to 
the category of the painted three-dimensional spatiality that regulates the reciprocal po- 
sitions and distances between objects, background, architecture and the human characters 
acting there. The perspective space has been made interactively explorable again, beyond 
the point of view chosen by the artist, thanks to augmented reality, which allows the viewer 
to enter the space portrayed in the pictorial image thanks to the three dimensions restored 
to the mural in the virtual space. Each mural has been provided with a thin horizontal band, 
containing three printed images designed to be “markers” for digital content activation. By 
scanning one by one each marker with the open-source app Artivive, different digital con- 
tent appear on observer's smart device: a short video clips narrating artist's life and poetry, 
an animated storytelling of the painted image and a three-dimensional model drawn ac- 
cording to parallel planes which, like a theatrical scenography, gives the perspective image a 
virtual three—dimensionality and an explorable spatial depth. 

Augmenting Angri won funding from the Open Call Street Art 2020, funded by the Embas- 
sy and Consulate General of the Kingdom of the Netherlands. lt was inaugurated in Angri 
on || October 2020 in the presence of the artists, university students, local authorities and 
the community. 
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REPRESENTATION CHALLENGES 
Augmented Reality and Artificial Intelligence in 
Cultural Heritage and Innovative Design Domain 


Evolutionary Time Lines, Hypothesis 
of an AI+AR—Based Virtual Museum 


Caterina Palestini 
Alessandro Basso 


Abstract 


The contribution related to the processes of knowledge and enhancement of cultural heritage, based 
on the potential offered by Artificial Intelligence and Augmented Reality, proposes an analysis of the sit- 
uation of L'Aquila's buildings ten years after the 2009 earthquake. The research, conducted on the basis 
of integrated surveys carried out before and after the earthquake, focuses on the application of AR/ VR 
devices through the implementation of Al. The aim is to propose a solution able to promote the use 
of the historical buildings of L'Aquila, activating at the same time the dynamics of cultural regeneration 
in the area, through the use of an App that uses AR+Al systems; a tour that can show with immediacy 
the urban architectural evolution, solving in part the intrinsic difficulties posed by the current situation 
of precarious visibility and accessibility of some noble buildings, subject to restoration, with the aim of 
highlighting the evolution of transformations, stylistic and structural changes produced by the different 
and stratified post-earthquake reconstructions. 


Keywords 
segmentation, heritage, machine learning, augmented reality, data libraries. 
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Fig. |. Development 
workflow, acquisition 
and use of data to 
build a deep learning 
model with Pytorch 
Vision, subsequent 
implementation on 
ARCore platform for 
editing the Augmented 
Reality SDK App. 


Introduction 


A recurring seismic history hangs over the urban context of L'Aquila, which after each cat- 
astrophic event finds the strength to rebuild its buildings, reworking the architecture often 
derived from the transformations produced by similar disasters in the past. L'Aquila's historical 
process of formation and change is in fact characterised by phases of renewal that can be 
traced back in some way to the historical sequence of the seismic events it suffered. The his- 
torical phases of the city’s evolution lead us to consider the dualism between natural events 
and anthropic responses as a guiding thread for analysing the events on which the constitu- 
ent moments of the urban context were formed and regenerated, specifically regarding civil 
construction. The reading carried out through the filter of the seismic recurrence defines the 
antecedent to investigate the metamorphosis of L'Aquila's buildings, to understand today’s 
situations considered in the pre- and post-earthquake comparison. Based on historical inves- 
tigations and surveys that integrate current data with the situation prior to the 2009 earth- 
quake, comparative representations are proposed to allow us to perceptively reconstruct 
the evolutionary time—lines of some historical Aquila buildings selected as exemplary models. 
The result is a documentary and informative material that, starting from the original project, 
makes possible to visualise the subsequent transformations, to appreciate at the same time, 
scrolling visually on a tablet or smart phone through an App that uses AR and Al systems, the 
design, the articulation of the facade analysed through semantic models, the current and past 
images that allow a dynamic knowledge of the artistic and cultural heritage under examination, 
proposing the monitoring and enhancement of a fragile architectural context, to be preserved 
and disseminated experimentally already said before (fig. |). 
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Machine Learning and Al, the Potential of an Evolving Technology 


The paradigms of cybernetic learning are progressively influencing, thanks to the tech- 
no-evolution of computing powers and a better accessibility to web networks, predictive 
models on the acquisition of notions through which man seems to approach ‘Data Science’ 
more and more easily, reinforcing the link between human and artificial intelligence in or 
der to achieve innovative approaches to cognitive insights. 

The Al, basically applicable when a machine imitates the ‘cognitive’ functions suitable to 
structure a logical system to ‘learn’ and ‘find solutions’, is nowadays employed in diversified 
application fields, domotics, intelligent writing, social computing etc., but we can identify 
its consistent support in the graphic—visual fields of Computer Vision and applicable in 
disciplines such as Geomatics and Architecture, where multidisciplinary studies, theoretical 
and practical, are slowly opening towards the analysis and dissemination of Heritage. In 
spite of these premises, in a purely visual environment it is highly destabilising to think that 


254 


Fig. 2. Programming of 
ARCore for Android and 
eventual conversion for 
ARKIT 4 for the Apple 
platform. Construction 

of time-line gallery of 
the historical buildings 
of L'Aquila (3D models, 
photographs, 2D 
drawings). 


a computer can generate ‘images’, albeit nowadays very unimaginative compared to human 
creativity, or superimpose almost imperceptibly for example one face on another Although 
the inevitable dystopian scenarios conjured up by the idea of cybernetic intelligences coldly 
replacing human minds instil a recurring doubt, as happens when any new technology is 
applied in the early stages, for now this suggestion remains a dispute without concrete 
scientific foundations, tempered by the benefits that artificial intelligence itself seems to offer 
today, as a support for an artificial logic that can better direct the more emotional human 
spirit towards concrete solutions. This is the specific application of a class of algorithms that 
automates the construction of analytical models by offering computers the possibility of 
learning in complete autonomy without being explicitly programmed. The use of algorithms, 
which learn through iterative procedures from data input, allows hardware to automatically 
identify operations to be activated without explicit programming. It is in such an environment, 
where augmented reality and artificial intelligence can collaborate effectively together, that it 
is possible to put algorithms together with models consisting of samples that can perform 
tasks autonomously without the need for complex programming steps. 

ML models use self-generated data from which patterns and correlations are learned, 
while AR is able to merge physical environments with digital content. Augmented Reality 
and Machine Learning, relatively young technologies that are actually shaping new opera- 
tional models, thanks to the large amount of available data, now seem to finally bridge the 
gap between the physical and the virtual world. 
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Creation of a Prototype for the Historiographic and Stratigraphic Contextualisation of 
L’Aquila Heritage 


The article proposes an application case study, still under development, in which the inte- 
gration between the two AI+AR technologies is experimented, proposing the use of an 
App aimed at promoting and enjoying the historical buildings of L'Aquila, activating trans- 
versally dynamics of tourist and cultural regeneration in the territory. 

Through the use of ARCore systems and the support of PytorchVision system on Faster 
R-CNN, an interactive tour via smartphone is defined that can show with simplicity and 
immediacy the architectural-urban evolution of the L'Aquila context. Partially solving the 
intrinsic difficulties posed by the current situation of precarious visibility and accessibility 
and providing an accurate visual time line of some noble palaces protagonists of the nu- 
merous restorations and superfetations, with the aim of highlighting the evolution of sty- 
listic and structural changes produced by the numerous post-earthquake reconstructions. 
The involvement of Machine Learning can currently be considered the best tool to speed 
up the development of Apps for the valorisation of Heritage, as it precedes automated 
processes to the creation of complex algorithmic codes, aimed at the recognition of ques- 
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tionable assets and functional to an easier loading in 3 


D space of explorable digital con- 


tents. The results of recent technological research have greatly increased the possibilities 
and relevance of virtual museology, visibly elevating it from the uses for real museums for 
which it was once intended. With the introduction of augmented reality, a navigable virtual 
reproduction is proposed in existing environments through the use of an app for smart- 
phones or tablets, configuring a new concept of diffuse virtual museum, which focuses 
mainly on the easier usability of a wider and heterogeneous public, together with a better 
enhancement and promotion of cultural capital that is in part precluded (fig.2). 


A virtual experience including three-dimensional reconstructions and interac 


ae 


tive insights 


makes available to the visitor assets that are not accessible or no longer exist, such as 
works that have been lost or remodelled over time. Moreover, the non-place in which the 


virtual museum is set up allows the materials of interest to escape from their spatial 


—tem- 


poral constraints, offering a time-line that provides a view of the conditions of the buildings 
in the various periods of interest, and to be accompanied by multidisciplinary in-depth 


analyses and thematic—bibliographical links that 


can integrate the superficial 


reading, re- 


sponding in a proactive and coherent manner to the needs of the users concerned. In the 


case study, these possibilities prove to be partic 
use of buildings affected by disasters or subject 
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ularly valuable in relation to the virtual ‘AR’ 
to heavy renovation, which are therefore 
they do not have a comprehensive, easily 
In general, visitors who interface with the 
ulting the works, operating in total freedom 


of observation and enlargement of the selected material, they are free to choose the level 
of detail and duration of the consultation, and can follow a suggested thematic path or 
create their own. AR thus constitutes a digital enrichment of physical reality through the 
superimposition of the former on the latter, providing the user with additional informa- 
tion relating to the surrounding environment. Unlike pure virtual reality, augmented reality 
uses devices that do not mask physical reality, so as to enable enhanced interaction with it 


through virtual technologies. 


This technology also makes use of more or less sophisticated tools necessary to create a 


correlation with physical reality, which gene 
as smartphones and tablets. AR tools do not require total iso 


rally consist of common portab 


e devices such 
ation from the surrounding 


environment, but their use presupposes the user's visual contact with them. This relation- 


ship is established thanks to a basic princip 
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source resource that, by providi 


developers to immediately use 
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coherent and recognisable way to the user The multimedia content consists of galleries 
of images documenting the historical evolution of the buildings and their changes in time, 
organised thematically or chronologically, through interactive timelines, in order to adapt to 


the needs of the user These can be supplemented by videos or audio clips that guide the 
reading of the buildings according to the segmentation inherited from Machine Learning 
models. The section devoted to digital reproductions also contains technical drawings and 
3D models of the buildings, arranged in a space that can be framed in perfect superim- 
position with the real buildings, which unfortunately are partly inaccessible and covered 
by scaffolding for safety purposes, useful for a more careful examination of their original 
globality or for a closer look at the architectural decorations and structural elements that 
are difficult to observe from street level. 


Conclusions 


The contribution outlines in an experimental way a methodological workflow developed to 
monitor and enhance the knowledge and the transformations suffered by the architectural 
heritage of the buildings of L'Aquila, inextricably linked to the seismicity of the places, to the 
typical fragility and resilience of the context. The resulting surveys, historical investigations 
and analytical comparisons have defined the basis for the organisation of the knowledge 
that, in the integration of AltAR technologies, has found the formula for optimising and 
using the data for dissemination purposes in possible new heritage promotion projects. 
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Abstract 


This paper aims to illustrate the first experiments deriving from a research project which intends to 
verify the potential of augmented reality technologies in the field of the valorization of food and wine 
heritage. In particular, a playful interactive editorial product was prototyped. It was made up of table 
board and picture cards to be consulted through an augmented reality application based on a web 
platform. The augmented contents were developed with the intention of showing a possible food 
pyramid built on the knowledge of the qualities of the typical foods of the Marche region. 
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food and wine, territories, experiences, board game, augmented reality. 
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Food as Cultural Heritage 


Food and wine heritage represents a fundamental driver that not only enriches tourist 
offer but also stimulates the economic, social and cultural development of a territory. 
For this reason, gastronomic tourism requires new strategic, economic and promotional 
models that encompass different but deeply interrelated areas [Garibaldi 2017]. 

t is no longer limited exclusively to the purchase of local products or the eating of typical 
dishes, but the activities also expand to participation in dedicated events or visits to farms, 
wineries, dairies, etc.A plurality of experiences in which the involvement of senses is qual- 
ifying, through which enjoying the cultural heritage of the place in an active and involving 
way, enriching the value of sharing. Food and wine therefore become a means for the 
recovery of collective memory and for the characterization of identity. 

According to the Rapporto Sull'innovazione Tecnologica nel Turismo Enogastronomico [Garib- 
aldi 2020], 62% of Italian tourists would use an application or a website which would lead 
to the discovery of the typical food and wine of the place, and 52% would like to visit 
production places which use multimedia technologies to enrich the visiting experience. 
The author of the Report reiterates that the use of technologically advanced tools and 
systems such as VR, AR, holograms or multimedia tables facilitate the relationship with the 
tourist, before, during and after the experience, stimulating a more immersive, engaging 
and personalized tourist experience. 
The research project Food and Wine Heritage in the Marche Region: Digital Storytell- 
ing Through Virtual and Augmented Reality [1], conducted by a multidisciplinary team 
composed of designers, architects, nutrition biologists and computer scientists from the 
University of Camerino, has set itself the goal of enhancing the Marche region through the 
use of digital tools and systems that exploit the potential of virtual and augmented reality. 
The project focuses on new narrative modes, in order to provide insights into the culture 
of food, and to tell and spread the variety of typical products of the Marche and the ter 
ritories to which they are linked. 

The reconstructions of the historical events of Marche's cuisine and its food traditions re- 
veal interesting relationships between popular and aristocratic, lay and conventual culture, 
but also between oral tradition and written codifications [Bellesi, Franca & Lucchetti 2010]. 
These researches gather a vast documentation that testifies how the territory of Marche 
already between the IX and Ill centuries B.C. at the time of the Piceni and then in the 
Roman age, was known for its cultivations and agricultural products, among which wheat, 
fruits, wine and olives.A great part of this heritage has been handed down through local 
traditions and in the diffused organic productions, today known in the world for the ex- 
cellence in wine production or for the primates in the oil and beekeeping fields and for 
the high quality of restaurants. 

However, enogastronomy in Marche represents not only an economic vector, but also 
an important socio—cultural factor, intimately connected to a complex set of material 
goods, made of architectural, artistic, environmental and landscape heritage, and immate- 
rial goods represented by culture, identity of places, ways of living and traditions, as well 
as human resources and entrepreneurial skills [Simonelli & Zurlo 2004]. 


Research Objectives 


The project, therefore, is based on the conviction that through visual design and the use 
of AR and VR digital technologies, it is possible to achieve the definition of innovative 
forms of storytelling capable of enhancing the local food and wine heritage and con- 
tributing, at the same time, to the revitalization of the cultural richness of the stories, 
traditions, know-how, beauty, widespread quality and the “genius loci” of the Marche 
region. In this sense, design constitutes the strategic lever through which to preserve 
the social and economic-productive characteristics of the territory, in order to avoid 
creating “synthetic” visit experiences, which can generally transform places into mere 
tourist attractions. 
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Fig. |. Graphic 
elaborations related to 
the maps of food and 
recipes of the Marche 
region (elaboration by 
Livia Barone). 


For these reasons, the research aims in particular to provide technologically innovative 
tools based on mixed reality systems and technically advanced devices and applications, 
able to spread the food and wine culture of the Marche region, telling stories and pecu- 
liarities of typical products, raw materials and food industry, and how these are combined 
with the territorial heritage and landscape. In addition, the technological models that are 
intended to be developed through VR and AR technologies are intended to promote 
interest in food and environmental education, enhancing not only local products but also 
the places of origin and production chains typical of the food and wine of the Marche. 
Basically it is foreseen the elaboration of a precise narrative strategy based on experiential 
activities, through the exploration of local itineraries and the discovery of quality food and 
wine products (when, where and how they are produced and consumed) [2]. 

In this scenario, in order to guarantee an organic and transversal development of the re- 
search project, a working group has been constituted. It includes and interconnects differ- 
ent disciplinary fields. In fact, designers and experts in digital representation were joined 
by biologists, nutritionists, scholars of the Marche region's diet and computer scientists. In 
other words, the components together constitute a framework of integrated and com- 
plementary competencies, extending from visual design to art history, from videography 
to landscape surveys, to computer graphics, 3D digital modeling, visual communication, 
interactive applications, database management, etc. 


AR Board Game 


Among the activities, those that explore the potential of AR refer to a system com- 
posed of a board game to be consulted through the use of an augmented reality ap- 
plication, to be used by smartphone or tablet, able to illustrate in an interactive way a 
possible “Marche” food pyramid, which aims to promote the knowledge of the qualities 
of the typical foods of the region. 

The board game offers the possibility to know the peculiarities of some local typicalities, 
through augmented contents, using a series of support cards that, as in an illustrated 
atlas, accurately describe the products both from a nutritional and a historical-cultural 
point of view. In essence, it is a system composed of tablet boards and cards to be 
consulted through the use of smartphones or tablets and an augmented reality app. 
This system makes use of a visual lexicon consisting of illustrations that faithfully re- 
produce ingredients and dishes, more or less known, of the local food and wine, made 
through a graphic synthesis that privileges a zenithal vision to allow the visualization 
of the food product in its entirety, and a chromatic system consisting of a palette of 
homogeneous colors that refers to the original ones. These representations are placed 
in a graphic field composed of a circular disc, which is meant to suggest the shape of 
the dish (fig. 1). 


While for the game board representing the geographical profile of the Marche region, 
neutral chromatic tones were used to obtain evident contrasts with the cards (fig. 2). 
The interactions are based on simple dynamics: the cards representing the single typi- 
cality, once positioned on the board and framed with your device, will allow the visual- 
ization of an animated infographic, which will return information related to the position 
that the food occupies in the food pyramid, a histogram that illustrates the nutritional 
values and the territory of origin located within the geographical profile of the Marche 
region, present in the game board. To complete the experience, a series of special cards 
dedicated to some typical recipes illustrate in the same way how to ‘correct’ their po- 
sition on the pyramid, and consequently the frequency of use, modifying dosages or 
types of cooking. 

The augmented content is composed of the illustrations on the cards and on the 
boards represented in 2.5 D mode, that is, they are two—dimensional graphics arranged 
in a three-dimensional space. The platform used is that of Artivive [3] and when the 
smartphone or tablet will frame the card positioned in the appropriate stall on the 
board, the cards, depicting the individual typicality, will allow the display of an animated 
infographic (fig. 3). 


Fig. 2. Marche in Tavola, 
table board. 


Fig. 3. Augmented 
information activated by 
the use of Artivive app 


262 


Conclusions 


Digital mediation techniques, 3D models, 360 panoramas, dynamic interfaces, redefined spaces 
and times of learning. Indeed, it is indisputable that today the ‘new’ media are the protagonists 
of a ‘shift’ towards renewed communication models that aim to an extension of the cultural 
offer in an increasingly rapid and immediate form. The communicative actions and the new 
forms of representation aim to facilitate understanding, to clarify aspects of complexity, to 
present concepts in a clearer and more concise manner, to make the information more ex- 
plicit and useful, while at the same time ensuring a high level of scientific content. 

In this framework, the research project explores the potentialities that can emerge from the 
integration of traditional communication systems made up of paper and editorial artifacts 
with technologically advanced tools based on mixed reality systems for the valorization of 
the excellences of the Marche region's food and wine sector Therefore, the board game 
project can become useful to understand, on one hand, the levels of interaction between 
paper and digital artifacts, and on the other hand, to establish the "balance lines of inter- 
action”, in order to maximize the general aims of the prototype. In fact, this integration 
between traditional and digital modes of communication must first of all stimulate the user 
to delve into topics both of a scientific nature, such as nutritional ones, as well as historical 


and cultural ones, which are combined with the territory. 


Notes 


[I] Project funded under the call established by the University of Camerino for the allocation of the 2018 University Fund (FAR). 
Duration: 24 months (1/2/2019 - 1/2/2021). 


[2 Among the products analyzed: olive all'ascolana, mela rosa dei Sibillini, miele dei monti azzurri, formaggi di Fossa, salame 
di Fabriano, pecora soprawissana, ciuascolo dell'alta marca, crescia fogliata di Fiuminata, torrone di Camerino, salame di fichi 
marchigiano, verdicchio di Matelica, Vernaccia di Serrapetrona, vino cotto di Loro Piceno, pesca della Valdaso, vincisgrassi, 
carciofo di Monte Lupone. 


[3] The website claims that: “Artivive is the Augmented Reality Platform for Art. This new technology allows artists to create 
new dimensions of art by linking classical with digital art.The digital layer opens the doors to a whole new world of possibilities". 
Artivive system is composed by two parts: Artivive App for the visualization and the Creational Tool where everybody can 
create digital layer to overlap to the reality. https://artivive.com (20 february 2021). 
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Abstract 


Nello Scille Cariddi is an immersive and interactive room, in which the marine environment of the 
Strait of Messina was employed as first experimental scenarios to develop and test the technology. 
The scenarios reconstruct different settings relating to this unique ecosystem and represent the 
outcome of a long process of research and systematization of audio visuals available at the Horcynus 
Orca Foundation. The room, therefore, transforms collections of films, integrating them with the 
production of specific images, into virtual environments containing structured catalogues and into 
interactive installations for educational, playful, scientific, and popular use. 
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immersive room, Scylla and Charybdis. 
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The Horcy 


nus Orca Foundation 


The Foundation is based in the homonymous interdisciplinary cultural park, managed by the 


same organ 


ization. The Horcynus Orca Park is named after the novel by Stefano D'Arrigo. Its 


activities involve a complex system of different fields of knowledge (literature, anthropology, 
marine biology, physics and chaos theory, natural sciences, archaeology, art, earth sciences) that 
compose the grammar and syntax of this millenary space: the scille cariddi. The Foundation 
thus represents an innovative bridge between creative languages, encounters between cultures, 
scientific research, technological innovation, experimentation with solidarity economies, ethically 
oriented markets, and dissemination of knowledge. The founding members of the Foundation 
are the Universities of Reggio Calabria and Messina, the CNR (National Research Council), the 


Ecosmed Research Centre and some socially and environmentally responsible companies. 


The Foundation involves researchers and scholars from different disciplines and has several offices: 
— in the monumental complex of Capo Peloro, north of Messina; 


— in Scilla and Reggio Calabria; 


— on the sea surface of the Strait, on the Kobold platform, the world's first experimental sta- 
tion to produce energy from marine currents, built through a project sponsored by the UN. 
Currently the main areas of the Foundation's commitment are three: 


— the pole 


of Mediterranean cultures, with the MACHO Contemporary Art Museum and 


the event of Mediterranean arts Horcynus Festival; 

— the International Centre on Marine and Environmental Sciences and Technologies, which 
operates under the auspices of the main UN agencies; 

— the Centre of scientific dissemination, cultural and educational tourism. 


Fig. |. Panoramic view of 
the “Oblò” scenario. 


The Research Paths and the Architecture of the Immersive Room 


The techno 


ogical development of the room was started as part of the project named Alfabeti 


dello scill'e cariddi, created by the Messina Community Foundation in partnership with the Horcynus 
Orca Foundation and co-financed by the Italian Ministry of Education, University and Research 


within the St 


ops the technology used in the first interactive cinema experiments. The technology used supports 
different media (photography video, sound) through interface logics. The system, therefore, inherits 


the formal g 
frame; on th 


tart Up Call Culture with increased impact. The experimentation elaborates and devel- 


rammar of cinema but it also evolves it, because it rejects the form and constraint of the 
e contrary, it expands the screen and integrates it with the architectural space. 
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The project, after a phase of pure research, focused on the construction of a prototype of great 
suggestion and attractiveness: the interactive immersive room, housed in the hall of the nine- 


teenth-century a 


Multichannel 


and the perfect synchron 
and homogeneous. By means of a recognition technology, vertical surfaces are 
the different ways of contact between the hand and the walls activate interactions in the 
video and sound effects. In this way, the audience constructs the projection, establishes its 


timing, and directs its development. 


+ 


rea (Bourbon fort) of the Monumental Complex of Capo Peloro, a space in 
direct continuity with the tour of the Horcynus Orca MACHO Contemporary Art Museum. 
emission systems are installed inside the room; thanks to the use of 3D models 
ization of the projections, the immersive environment 


+ 


is smooth 
touchscreen; 


For the realization of the prototype room the following devices/software/systems were installed: 
— a cluster of 8 video projectors, each connected to an infrared sensor and to as many 


computers p 


lus a “control” machine; 


— proprietary software intended for the automatic generation of masks to deconstruct 
the original videos and extrapolate the moving subjects which are then recombined in the 
recomposition of the immersive scenarios; 
— a control system via internal hotspot, connected to the network for on-site control and 
remote system maintenance. 


Fig. 2. Detail of the 
“Meduse” scenario. 


The technological structure was designed in relation to the size of the room and the range of 
action of the sensors, to achieve complete coverage of the projection and interaction on the en- 


tire perimete 
shielding mas 
the video sce 
The device a 
setting 16 to 


graphic prod 


r of the walls. Through video-mapping techniques and the use of specially designed 
ks on the architectural elements of the room, it was possible to better integrate 
nic effects and the perception of the historical-monumental value of the building. 

lows to preset the number of simultaneously detectable touches (in the current 
uches for each of the 8 clusters). T 
the interactions to use the data streams in the render engine. The audio, the reading and the 
process of flows generated by the sensors, the | 
uction are processed on different hardware but unified by a standard communi- 
cation protocol (OSC — Open Sound Contro 
the workload is managed in a cluster for better 


he system reads and passes the positions of 
nteraction logics, and the multi-channel video 


protocol) which guarantees synchronization; 
system performance. 
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An advantage of the component system is that the configuration can be quickly changed as 
required. For example, it is possible to concentrate all the components in the same hardware 


Fig. 3. Detail of the 
“Meduse” scenario. 


or, in output, to compose a cluster of computers by dividing the workload and thus potentially 
expanding performance indefinitely. Moreover thanks to its modularity, the developed technol- 
ogy is particularly versatile. The number of clusters that can be integrated into an infrastruc- 
ture of this type has no predetermined constraints; this makes it possible to vary the arrange- 
ment of the individual modules and therefore the shape and overall size of the installation. So, 
it is possible to vary the number of screens (and video projectors), the number of sensors 
and computer vision algorithms connected to the system, the number of speakers and audio 
signals, depending on the type of interactive environment and the place where it is installed. 

An Ambisonic audio system was installed in the room with the binaural technique capable of replicating 
the functioning of the human auditory system. This technique allows to listen to tracks with an unprec- 
edented quality, virtually bringing the viewer into the sound field in which the recording was made. 


The Software Architecture of the Immersive Room 


The software is structured according to a modular architecture. The individual modules are 
specialized, i.e. each one has a specific task. Furthermore, each module can communicate 
with the other modules via the aforementioned Open Sound Control protocol, which 
allows the implementation of additional modules without modifying the existing ones. The 
control computer manages the installation by receiving sensor messages at the input and 
sending the display commands at the output or by performing sound effects and back- 
ground audio. The modules are distinguished by the following functions: 

— Sensors (e.g. Kinect, Realsense); 

— Logic; 

— Rendering; 

— Room management. 

The Sensors allow users to interact with the room. For each sensor there is a specific mod- 
ule that manages it and makes it able to communicate with the room logic. 

The Logic consists of a program that receives the sensor messages, processes them accord- 
ing to the interaction logics and sends the commands to a rendering engine. The logic of 
interaction depends on what we have defined as scenarios. Each scenario describes the set 
of visual and sound events that at a given moment arise from the interaction of users and 
the mechanisms that regulate them. The Rendering engine is the component that takes care 
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of visualizing the contents and effects of user interactions. Basically, he is a mere executor 
of the commands given by the control computer The overall Management of the room is 
carried out by a module called manager”, whose task is to switch the entire room on and 
off, and to start and stop individual programs when necessary. Manager also provides a web 
interface for changing scenarios and switching the entire room on and off. 

The only module that controls the status ofthe room and the active scenario at a given time 
is the one that manages the logic. The sensor modules and the rendering module have no 
knowledge of the room status but are limited to executing commands (rendering module) 
or sending information regarding user interaction (sensor modules) to the room logic. 


The Prototype and Research Developments 


The first prototype contains several immersive scenarios of the marine environment of the 
Strait; they are the result of a path based on research, organization, metadata, and optimiza- 
tion of archive resources. 

One of these scenarios reproduces from the inside the walls of a wreck with portholes of various 
shapes within which it is possible to reproduce films from a collection relating to underwater ar 
chaeological sites in Sicily, made available by the Superintendence of the Sea of the Sicilian Region. 
Other scenarios allow the public to discover the environments and marine species of the 
sea surface and the abysses of the Strait; thanks to the interaction, each touch makes sea 
creatures appear alive and in motion, selecting them randomly from a database. 
Nello Scill’e Cariddi is a prototype dedicated to underwater scenarios, but the project makes 
it possible to stage, at low cost, settings relating to different themes: figurative art, virtual sce- 
nography for performing arts, archaeology; scientific dissemination projects, sensory itiner- 
aries for disabled people and, more generally, educational paths. The Horcynus Orca Foun- 
dation has already created additional collections of interactive immersive environments and 
therefore the archive is constantly being implemented. 
The first interactive art installations have already been created, including Ultrathinking by the 
Iraqi artist of Kadir Fadel and La Habana Sobre Ruedas by the Cuban collective composed 
of Claudia Hechavarria Segura, Juan Alberto Matamoros Nues, Elvys Ariel Urra Moreno; 
a first experimentation of ensemble music in which it is possible to activate with infinite 
combinations up to 32 instruments of a virtual orchestra curated by Luigi Polimeni; the first 
experiments of interactive virtual scenography through the collaboration with Mana Chuma 
Teatro. Some virtual scenarios on matter and the cosmos are at an advanced design stage, in 
collaboration with the National Institute of Nuclear Physics; titles suitable for children with 
particular types of autism spectrum disorder, in collaboration with the CNR-IRIB (Institute 
for Research and Biomedical Innovation) of Messina; a historical-archival project for the 
reconstruction and investigation of the scene of the Portella della Ginestra massacre. 
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Abstract 


In the digital age, museums have seen a significant expansion of the spaces and tools for disseminat- 
ing knowledge. Gallerie dell'Accademia had commissioned a permanent and multimedia video that 
summarized the results of the interdisciplinary research developed within Visualizing Venice — S. Maria 
della Carita, S. Agnese, and the Gesuati, three insulae make one — that told about the history of the 
urban and architectural transformations of the insula. Since it is not a totally immersive installation, 
some years later Gallerie asked to make some changes to the previous animation to allow a greater 
interactivity between user and observed data. In order to keep the visitor's attention threshold high 
and active, different sections of short duration have been introduced, which can be interrogated and 
activated by sensors, while a Sound Shower system facilitates the understanding of the multimedia 
story through the listening of a narrator's voice. 


Keywords 
virtual reality, digital model, database, interaction, immersivity. 
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Venice in Motion: the Site of the Gallerie dell’Accademia Between New Media and History 


The idea of Venice as a city constantly in motion has been recognized for some time now, so 
much so that it has become a truly significant feature of its own image. Movement does not 
only concern individuals, goods, and objects, but also its urban and architectural space. This 
pertains to the transformation of buildings and the relationship between land and water, 
but also — and equally importantly — it involves the functional use of space, the articulation 
of collective facilities, and the city’s appearance. To these extents, the area of the Gallerie 
dell'Accademia represents an exemplary case study of inquiry. Located on the southern 
margin of Venice, in a transitional space stretching between the Grand Canal and the Gi- 
udecca Canal, the site consists of three insulae named for the ancient religious complexes 
that once moulded its space: Sant'Agnese, the Gesuati, and Santa Maria della Carita. The 
atter, which consisted of the church, the adjacent monastery of the Lateran Congregation 
of the Canons Regular of Saint Augustine, and the Scuola Grande, was converted at the 
beginning of the nineteenth century to accommodate the Academy of Fine Arts and later, 
in 1817, the Gallerie dell'Accademia [Codello 2017]. The history of these institutions and 
pertinent buildings has been widely investigated by art and architectural historians who de- 
voted particular attention to Andrea Palladio's design for the renovation of the monastery 
of the Carita [Modesti 2005, 2005b]. However, the long-term history of today’s site of the 
Gallerie dell’Accademia and its interwoven relationships with its surrounding urban fabric 
have remained largely unstudied, in part because of the difficulty to show the public how 
the museum was inserted into existing spaces and in what way these places have changed 
over time [Svalduz 2017]. 

If looking today from above, the area emerges as a compact and uniform site, but actually 
its topography is the result of a process of urban and architectural stratifications that spans 
a thousand-year history. Until the beginning of the nineteenth century, the settlement was 
designed by an articulate system of waterways, which included two straight canals — the Rio 
della Carità and Rio di Sant'Agnese — crossing the site longitudinally and flanking both sides 
of the Lateran Canons’ complex. As part of the process of modernization undertaken in 
the aftermath of Napoleonic rule, the two watercourses were filled in, while the erection 
of a new bridge over the Grand Canal definitely brought the entire area out of isolation by 
ensuring a rapid pedestrian link to the rest of the city [Galeazzo 2018] [1]. 

These changes profoundly altered the urban settlement's configuration, as well as the visual 
perception of the ancient buildings, especially of those belonging to the Lateran monastery. 
What had previously been a sort of fortress surrounded by water gradually became inte- 
grated into the city [di Lenardo 2013]. In addition, the conversion of religious buildings into 
secular places eventually led to an invasive campaign of renovation of the ancient structures, 
which were stripped of their decorations to avoid any visual-mental connection to the 
former ecclesiastical complex. 
The emerging dialogue between new media and traditional historical research offered the 
multidisciplinary team the option of experimenting an innovative approach to visualizing and 
communicating these historical changes. The video ‘The History of a Site and its Transfor 


+ 


mations: From Carità to the Gallerie dell’Accademia’ was created to exploit the affordances 
of emerging technologies as a powerful tool for giving visitors a simplified and compre- 
hensive understanding of the area's elaborate changes over the centuries. The project had 
two overlapping objectives: to shed light on the urban and architectural transformations of 
the urban site while focusing on the impact these modifications had in the land — and wa- 
ter — based Venetian traffic systems and to visually represent the relationship between the 
previous structures and the present spaces of the museum to help visitors understand the 
origins of the complex. The philological reconstruction of this large section of the city was 
based on the collection of a broad scope of historical sources and identification of a series 
of significant chronological phases that represent many temporal snapshots in the history of 
the insula's transformation. These span over five centuries and help visualize the tight bond 
between space and architecture. Beginning with the renowned bird's—eye view of Venice by 
Jacopo de’ Barbari (1500), the video then reconstructs the eighteenth-century shape of the 
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newly—built church of the Gesuati as well as the spatial organization of the monastery of the 
Carita through both cartographic and iconographic sources. These encompass the first city’s 
map of Venice by Ludovico Ushi (1729), several detailed drawings executed by local magis- 
trates, a sequence of Canaletto's paintings, as well as some precious surveys elaborated by 
the proto Pietro Angelo Fossati immediately after the suppression of the Augustinian mon- 
astery in 1793. Napoleonic (1808-1811), Austrian (1838-1842) and Austro-Italian (1867- 
1913) cadastral maps were instead crucial for mapping nineteenth-century transformations 
of the waterway and pedestrian system [Galeazzo, Pedron 2014]. 
While the goal of the video is primarily didactic and informative, showing the transforma- 
tion of the area from the early Renaissance to the twentieth century, it also allows users to 
physically experience the transformation of water-courses into pedestrian paths through 
a virtual trip along the now filled—in canals, it brings also a contextualized understanding of 
how an amphibious city like Venice was subjected to a gradual process of land reclamation. 


Digital Representation of Architectural and Urban Changes Overtime 


Cinema and photography embodied the main cultural interfaces of the 20th century, tools 
through which it was possible to describe the movement. With the advent of digital rep- 
resentation, virtual reality has ‘stolen the scene’ from previous forms of communication. 
According with other devices, the monitor has become more and more important since 
it constitutes the limit of the virtual vision, the tool with which the user/observer must 
interface an illusory space through a sort of virtual window. It is a threshold from which 
to experience a mosaic of information perceived from multiple points of view referring to 
many different reading planes. The main novelty of the man-computer-culture interface lies 
in the revolutionary way in which machines present data and allow to interact with them 
[Manovich 2009]. 

The installation created for the Gallerie dell'Accademia involves the projection of a digital 
movie on a videowall [2]. The first frames of this movie serve to contextualize the three 
ancient insulae as they were in the |6th century urban fabric. The digital reconstruction 
was carried out through a process backwards in time starting from the actual state of the 
insula. The narrative structure and the storyboard of the installation follow instead a linear 
exposition of the events, which allows a reading without interruptions and reversals the 
chronological order of the events. The story retraces the historical phases starting from 
1500 up today. The large amounts of documents, historical data, lit 

constitut 


terary and graphic sources 
te the documentary substrate on which this work is based. 
The starting point of the modeling phase is the state of the insula as we did in 2015, when 
the original movie was created. Maps, engravings, drawings, surveys, chronicles, photographs 
and publications were added to the virtual platform. 
The next step concerns the identification of the historical moments in which the most sig- 
nificant transformations of the urban fabric occurred, identifying six main historical phases 
summarized here and appropriately arranged with historians, which provided the starting 
dates: 1500, 1729, 1794, 1811, 1842, 1906 up to 2015.Where the information on individual 


buildings was not sufficiently clear and comprehensive to develop a det 


tailed model, we 
preferred to omit its representation, opting for a volumetric abstraction of the building 
rather than originating an arbitrary interpretation, not very rigorous from a scientific point 
of view. In the movie, the concept of time is explicitly expressed through a timeline, always 
visible on the screen, which attests the chronological sequence of events throughout the 
centuries considered. The timeline, arranged vertically, determines the page layout. Moving 
forward chronologically, it ideally cuts the screen into two parts (one twice wider than the 
other), following the horizontal division defined by the width of the monitors that make 
up the video wall. The size of the monitors and their mutual approach define the standard 
layout, the guidelines of an ideal and a modular grid, that help us to insert the documents. 
On the right, the visitor can see the plan of the insula, a static image accompanied by some 
tags that highlight the main urban and architectural changes occurred over time; on the left, 
instead, all the documents appear in a fading succession (fig. 2). This documents are the data 
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Fig. |. Layout of frames of 
digital video. 


that allowed us to model the urban situation in a precise historical phase. This roundup of 
static images concludes with a render of the digital model at the specific year To interrupt 
the static images of the movie we set a virtual tour around the insula, as it was in | 794, along 
the canals now filled-in. The animation is introduced by a plan that shows the path of the 
camera around the insula della Carita during the tour Suddenly the timeline, moving to the 
right side, gives the space to a full-screen movie: a representation within the representation, 
in which the viewer is involved in the perception of urban spaces from a today unusual and 
no longer available perspective that coincides with that of a visitor who is navigating the 
canals using a gondola, exactly as a citizen of the past centuries. The camera, simulating the 
eye of the observer, moves through two canals — the Carita and Sant'Agnese — in a position 
slightly above the water level, stabilizing a perspective horizon that coincides with a line just 
a bit over the walking surface. The movement of the camera inside the virtual environments 
is necessary to overlap a document directly on the 3D model to explain the reconstruction 
of some elevations of the buildings that face the canals, of which we have a precise docu- 
ment, like the precious surveys made by the engineer Pietro Angelo Fossati in 1793. These 
drawings can be rightly positioned only in a virtual 3DF model that takes into account the 
movement of an observer floating on a boat. Following the idea of updating our first installa- 
tion to new technologies, the same team of scholars has recently proposed to the museum 
an integration of the display with other in—depth historical studies to offer visitors a broader 
knowledge of the site. Users will be invited to choose between different lightening chapters, 
which are diversified basing on analysis, topic, and disciplinary approaches [3]. 


Conclusions 


Recently the Gallerie dell’Accademia has asked for some changes to the previous animation, 
the aim was to improve the interactivity between user and data. At the moment the ad hoc 
app, is a loop animation. But in the next future using an interactive menu by a gesture system 
it will be possible to chose which kind of documents it is going to visualize: digital videos, 
images or texts. The app keeps in connection the input and output devices linked to the PC 
in order to allow to check how to visualize documents. 

The Kinect's cameras allow to the user to interact trough e gesture system using only 
movements by hands and arms, without using fisical devices that control the movements. 
The open hand pointed to the monitor will allow to activate a main menu, moving on the 
differente voices and choseeing the preferit section. In the same waythe user will come 
back to the principal animation. Finally the sound shower system will allow e directional and 
localized diffusion of audio track of videos, in order to obtain a more immersive experience. 
This tool conveys to the sound in a limited area of the room not to disturb the visitors in 
adjacent exhibition spaces (fig. 3). 
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Fig. 2. Devices of the new 
interactive installation. 
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The introduction of an interactive action makes between visitor and installation made the 
reading and the learning of the urban and architectural story of the of the insula della Carità 
more involving. Operations of this kind can be considered as alternative tools, available to 
museums, for a transversal dissemination of knowledge, but this implies that the role of the 
representation of architecture and the city is called to respond to new challenges [4]. 


Notes 


] The Carità and Sant'Agnese canals were filled in 1817 and 1863 respectively. In 1854, the English engineer Alfred Henry 
eville built a suspended iron bridge, which was replaced in 1933 by a supposedly ‘temporary’ wooden bridge by Eugenio Miozzi. 


N 


[2] Each liquid crystal monitor is 46' in 16:9. The assembly of the devices provides for a 3x3 arrangement, i.e. three monitors 
along the horizontal direction and three along the vertical one set on one of the longitudinal walls of room 4, occupying a 
al area of 5.238 square meter The realization of the video assembly was achieved thanks to funding and sponsorship from 
Samsung Electronics Italia s.p.a. and Ventian Heritage, which supplied the monitors necessary for the projection of the images. 


O 


pei 


[3] These narratives will include: the investigation of the insula dell'Accademia as an area between various ecclesiastical 
communities; the digital reconstruction of Andrea Palladio's design for the monastery of Carità and its relation with the 
realized buildings of the museum; and the nineteenth- and twentieth—century architectural transformations of both the 
exterior and interior spaces of the Lateran Canons' complex, made to house the Academy of Fine Arts and the Gallerie 
dell'Accademia. A specific chapter will also be devoted to the campo (square) that faces the Grand Canal and its ancient 
functions of cemetery, docks for boats, place of humanistic encounters, space for ceremonies and events. 


ES 


[4] Venice in Motion: The Site of the Gallerie dell'Accademia between New Media and History was written by L. Galeazzo 
and E. Svalduz; Digital representationo of architectural and urban changes overtime was written by I. Friso and C. Monteleone; 
Conclusion was written by F. Borella. 
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Wood Inlays and AR: 
Considerations Regarding Perspective 
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Abstract 


The contribution reflects on the options provided by recent Augmented Reality (AR) applications to 
the knowledge and enhancement of a nationally and internationally renowned cultural asset: perspec- 
tive wood inlays. 

The methodology envisages two closely—connected stages initially involving perspective decoding and 
ensuing reconstruction of the illusory model, followed by the creation of a set-up for the AR pursuant 
to the digitalization of the three-dimensional model. 

In AR, perspective is the ideal tool to virtually experience the space represented in the decoration; at 
the same time it provides the most suitable solutions required to optimise the AR project. 


Keywords 
wood inlays, perspective, augmented reality, immersivity, dynamic perception. 
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Fig, |. Schematic drawing 
reflecting on the possible 
relationship between 
objects and the picture 
plane in perspective 
construction (left) and 
the model appearing in 
front of the target in AR 
applications (right). 


Introduction 


This contribution [|] focuses in particular on wood inlays that primarily use a perspective ap- 
proach to create the image, i.e., early Renaissance and sixteenth-century intarsia that exploit 
an elegant perspective technique to produce wood decorations. 

There have been many recent examples of AR/VR applications used in environments embel- 
lished by wood inlay decorations. In particular the ones in Federico da Monfeltros studiolo 
in Gubbio and Urbino which were developed to virtually recreate the rooms’ artistic quality 
[2]. Said studies focused more on communicating the cultural asset rather than speculating 
on the perspective-geometry used to create the illusory space, achieved due to the sfonda- 
mento of the plane of the wooden cladding. 
This contribution will instead concentrate on the common projective origin of the two rep- 
resentative models: on the one hand the perspective model, on the other the digital model 
behind the AR experience that can be visualised on an ad hoc display. 
Our goal regarding the projective and perspective features linking the perspective sfondato 
with tts AR version is to verify whether these features are the key elements required to suc- 
cessfully recreate the spatial complexity of this particular artistic genre. 

To test the methodology we selected an intarsia made by Brother Damiano Zambelli 
around the year 1530 and inserted in the backrest of the choir stalls in the presbytery of 
the Basilica of San Domenico in Bologna (opening image, left). When an onlooker looks 
at this intarsia, or at any other perspective inlays, he perceives an illusory space beyond 
the wooden frame, in this case a quadrangular hall with a Lombard-style coffered ceiling 
resting on two rows of three pilasters, the remains of a building in the middle ground, and 
a temple in the background [3]. 


Considerations Regarding Perspective and AR 


Two features betray the similarities between perspective and AR: one involves the funda- 
mental elements of projection, the other concerns the elements participating in the creation 
of the perceptive experience of amplifying space. 

The first step is to verify parallelism between the centre of projection, the picture plane, and 
the model to be represented as crucial elements of the perspective, and, respectively, the 
virtual camera, the target, and the three-dimensional digital content in the AR. 
The effect of amplifying real space is created by skillfully using the perspective technique; it 
is effectively similar to the AR experience after accurate correlation is established between 
perspective construction and the design of the virtual fruition application. 
The illusory space conjured up in the wooden intarsia is possible thanks to the observer's 
sensitivity and visual and artistic knowledge, while the perceptive functioning of the AR is 
generated by superimposing suitably designed digital contents on real space; these contents 
are activated by establishing a specific target. 
In this respect, we must carefully consider the element to be used as a target to recognise 
the virtual model that we can presume to be either the inlay itself or an orthorectified image 
of the inlay itself. 
As we all know, perspective theory allows for endless positions of the model to be rep- 
resented compared to the picture plane (in fact the object can be between the centre 
of projection and the plane, beyond the plane, or even astride the plane). Nevertheless, 
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Fig. 2. Concept view, 
inside the choir of the 
basilica of San Domenico 
in Bologna. The image 
shows user perception 
of the application of AR 
in the case the necessary 
changes are not made to 
obtain the perspective 
sfondamento. 


when we look at a wooden intarsia the perceptive effect sought after and recreated by 
the artist is to place this space ‘beyond’ the physical limit of real space, i.e., beyond the inlay, 
which, in turn, can be equated to the perspective picture plane (fig. |). Instead in AR, after 
activating the application the digital model appears in front of the target (fig. 2). In per- 
spective intarsia applications this disorients the onlooker who no longer easily perceives 
the continuity between the two-dimensional perspective image and the reconstructed 
space in the digital model (fig. 2). 

To restore the immersive effect a solution could come from the nature itself of the wood 
inlays and their dimensional characteristics. In fact, inlays are primarily framed by a scansion 
of the wood surface so that the dimensions of each sfondato corresponds to that of an 
opening (door or window). The space imagined beyond the wooden plane appears to be 
divided into as many sfondamenti as the number of perspective panels, usually bordered by a 
compositional element that acts as a frame for the intarsia. This allows us to theorise that the 
latter is the element tasked with mediating the shift from the perspective image to its digital 
reconstruction by inserting, in the model, an element present in the target. 


Lets now focus on the case study, i.e., the inlay in the choir stall in the Basilica of San Domenico. 
We have outlined the aforementioned considerations because in AR the digital model is 
visualised in front of the target; the chosen target is in fact the intarsia itself, while the one 
perceived by the onlooker is a model that does not ‘pierce’ the wooden plane, but instead 
exits it and is projected forward into real space, nullifying the illusory effect of spatial depth 
beyond the inlay. To eliminate this undesired effect, a decision was taken to make part of 
the target (i.e., the frame around the scene) an element of the model produced for the AR 
application. The outcome was a virtual frame perfectly superimposed on the physical target 
in front of said urban scene. 

Solving the issue of the target is the first step in achieving successful restitution of illusory 
space. Another inconvenience is the presence of elements in real space that conflict and are 
visualised with the ones in the digital model — possibly including the target itself. 

Once again, the solution was found in the common projective origin of the wooden sfondato 
and AR application. In this case, the applied strategies were instigated by the perspective 
used in theatrical stage sets and photography. 
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Fig. 3. Box set (top left) 


(top right) applied to the 
virtual model (bottom). 


A box-like environment inspired by photographic box sets was created to isolate the urban 
scene beyond the frame and stop the real environment from being considered as a back- 
ground. This involved creating a delimited digital space characterised by neutral textured 
materials in which the rounded corners did not reveal the change in position of each plane. 
Inserting the digital model in this box eliminates the presence of real space elements that 
are thus inserted in virtual space, helping to reinforce the perceived effectiveness of the AR 
experience (fig. 3, left). 

Nevertheless, the digital model and the box set in which it is inserted are both a certain 
size and can be visualised as an insertion in real space; this is the third element that helps to 
weaken the illusory sfondato of the wooden surface. Once again the solution lies in the per- 
spective-scenographic origin of the two models, the perspective model of the intarsia, and 
the illusory AR model; it is reminiscent of the proscenium arch in many theatres or mobile 
stage sets. When a foreground frames the scene it amplifies the perceptive effect of depth; 
this has been common knowledge ever since antiquity, especially in the theatrical world; the 
‘trick’ is used both in the field of perspective and that of photography. 

In a theatre, the proscenium arch in the foreground plays a dual role: apart from framing 
the stage and interrupting continuity with real space, it makes the space of the stage less im- 
mediately ‘measurable’ and isolates the parts of the scenery and machinery that spectators 
should not see. 


La 


Placing this element in the model elaborated for the AR application involves inserting a sort 
of screen and assigning it certain characteristics and materials so that it isolates the model, 
but cannot be seen when the application is used. In our case study, for example, this addi- 
tional insertion of a screen in the foreground, was appropriately cut to show only the part of 
the sfondamento that the onlooker was allowed to see (fig. 3, right). 

The presence of an element of the target in the digital model, coupled with the creation of 
the box set and insertion of a screen, facilitated seamless exploration of the space of the 
AR with fruition of the real space where the inlay is situated; it also facilitated a comparison 
between the digital model and the perspective image. 
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Conclusions 


The use of AR in perspective applications and, in particular, in wood inlays, raises several 
questions; a suitable solution lies in the realisation that the two systems share a projective 
origin. Bearing this in mind, and focusing on the world of theatrical stage sets and photo- 
graphic technique — two technical—artistic mediums with the same roots and same scientific 
evolution — provides us with important knowledge and helps solving some of these issues. 
Obviously this contribution ignores other aspects that are nevertheless crucial and should be 
considered in a much broader study. 

In fact, tt will be necessary to tackle the question of the implementation of materials, lights 
and texture in the model for the AR application; these elements can be identified by analys- 
ing the inlays themselves, but will have to be correctly managed. 

Another aspect not tackled here, although inevitable, is the relationship between the posi- 
tion of the observer of the perspective (presumed to be stationary in the environment) and 
the kinetic nature of AR exploration. 

Nevertheless, once again, in the case of perspective inlays, AR proves to be the right tool to 
enrich the experience of all those who wish to enjoy cultural heritage thanks to an approach 
which, by increasing perceptive options, also includes scientific contents — all too often rele- 
gated to the back burner. 


Notes 


[1] Although all the authors participated in the whole research, the study of the wooden intarsia, their perspective construction and 
specific case study was performed by M. Fasolo; the part regarding the relationships between perspective and AR was performed by 
L. Carlevaris; in-depth operational studies and the creation of the models and contents of the AR application were performed by F 
Camagni. The conclusions are part of the joint research project. 


[2] Regarding the studiolo in Urbino, see: Roberto Mantovani, <https://www.youtube.com/watch?v=kShbXHY17GO0> (consulted on | Ith 
February 2021); the project Lo studiolo di Gubbio: tour virtuale e ipotesi ricostruttive di un microcosmo umanistico; Paolo Clini, Lo Studiolo 
di Federico da Montefeltro. Fruizione del Cultural Heritage attraverso nuove tecnologie di realtà immersiva. <https/wwwfacebook.com/ 
watch/?v=690533685 | 14053> (consulted on | Ith February 2021). Regarding the decorations in the studiolo in Gubbio, currently housed 
in the Metropolitan Museum in New York, <https:/Awww.youtube.com/watch?v=8M TgRIEUHBg> (consulted on | Ith February 2021). 


[3] Cfr. Alce 2002. 
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REPRESENTATION CHALLENGES 
Augmented Reality and Artificial Intelligence in 
Cultural Heritage and Innovative Design Domain 


Augmented Reality and Museum 
Exhibition. The Case of the Tribuna of 
Palazzo Grimani in Venice 


Giuseppe D'Acunto 


Abstract 


The purpose of this research was the exploration of the potential of the combined use of augmented 
reality and rapid propotyping for the enhancement of Cultural Property. This specific study explored 
how these innovative methodologies can be effective in the creation of museum exhibitions which 
are not only able to show the contents of the exhibition itself in an original and captivating way, but 
also to recover the memory of a place by reconstructing the original position of the surroundings 
strictly connected to the sculptures they have been deprived of over the centuries. 

It is the case of the Tribuna of Palazzo Grimani in Venice and of the statues that used to adorn its 
walls. The majority of these statues are now preserved in the National Archaeological Museum of 
Venice in an attempt to reconstruct their original appearance and disposition, and to offer the public 
a wholesome vision through augmented reality. 


Keywords 
augmented reality, digital models, digital survey, museum exhibitions. 
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Fig. |. (left) 
Reconstruction of a 
portion of the Tribuna by 
Federico Zuccari, 1582. 
British Museum, London. 
(right) Reconstruction 
of the wall facing the 
entrance. Favaretto l., De 
Paoli M. 2010. 


Among the various challenges that those who deal with exhibitions have to face every day, 
one of the most stimulating and insidious at the same time is the representation of the in- 
tangible value of the exhibited goods. Representing the intangible stratification of meanings 
and contents of a fragment of heritage — it being archaeological, architectural, cultural, or 
of any other nature — is an undertaking not to be taken for granted and is characterised 
by the need to maintain a continuous equilibrium between scientific rigor and narrative 
effectiveness. From this point of view, the evolution of the narrative codes and the increas- 
ingly widespread and rooted presence of the media have played a fundamental role in, on 
one hand, enormously enriching the range of solutions available to designers and curators, 
and, on the other hand, making this field significantly more complicated. Therefore, it is not 
surprising that, with usable patterns increasingly more shaped by the indissoluble influence 
of technology and its complicated rules — immediacy, accessibility, superabundance, experi- 
entiality — the contamination of the traditional codes with innovative languages and other 
narrative solutions has become a trend that is as natural as unavoidable and necessary. This 
is even more evident when museums are concerned, since the quality of an exhibition is 
determined, among other factors, also by the ability to show the intangible value of the 
exhibited goods in a tangible, effective and, most importantly, universal way. 

The reconstruction of the position of the statues of the Tribuna of Palazzo Grimani in Venice 
exegetically explained in this essay is an experimentation of a museum exhibition in one 
of the most iconic places of Venice structured and organised according to the possibilities 
offered by augmented reality. The purpose is to show the original aspect of this place that 
has been lost over time. 


The Tribuna was built in the second half of the | 6th century at the will of Giovanni Grimani, 
patriarch of Aquileia. It was clearly inspired by the Pantheon and illuminated from the above 
by an opening in the centre of the roof vault. lt contained an entire collection of statues in a 
very scenographic way due to an articulated system of lights and a refined system of niches 
and shelves. Even though there is no written evidence, it seems that the critics have en- 
dorsed the theory according to which Francesco Sansovino curated the architectural proj- 
ect of the Tribuna, as reported for the first time in a small anonymous guide of the end of 
the | 8th century called Pitture e scolture nel palazzo di casa Grimani a Santa Maria Formosa. 
This space was attributed the name ‘Tribuna’ only in the descriptions of the guides between 
the 18th and 19th centuries [1], but it was originally known as Antiquarium (studio of antiq- 
uities), as reported by Francesco Sansovino in his famous book ‘Venetia citta nobilissima et 
singolare’ dated 1581 [2]. In another passage of this book, there is a detailed narration of the 
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Fig. 2. The screen of the 
application in augmented 
reality with an example 
of multimedia card with 
the in-depth interactive 
contents for each 
individual statue. Digital 
elaboration by Valeria 
Sambucini. 


visit of Henry Ill, king of France, and Alfonso II, duke of Ferrara, to patriarch Grimani in the 
autumn of 1574. Sansovino described the amazement of the two illustrious guests towards 
the wonders that the Venetian Palace and, in particular, the Tribuna — where the entire col- 
lection of statues was displayed — offered. 

Thus, it can be assumed that the Tribuna was largerly complete between the 60s and 70s 
of the | 6th century, and the fame of its wonders had already spread throughout Europe, 
even though this setting, as originally organised, survived only less than thirty years. While 
still alive, Grimani bequeathed the entire collection of statues to the Serenissima Republic 
of Venice in 1586, with the intention to create a Public Statuary situated in the anteroom of 
the Marciana Library. According to the chronicles of the time, the transfer of the statues to 
the Statuary of the Serenissima started in 1593 (the same year in which Grimani died) and 
ended in 1596. During this operation, the first inventory of the entire collection, known as 
the Pellegrini Inventory [3], was compiled. This document is still of great importance in the 
attempt to recreate the historical issues of the Tribuna and it is one of the few written de- 
scriptions of the entire collection of statues in the room before their transfer to the Public 
Statuary. Another fundamental document that gives us the precise idea of the original as- 
pect of the Tribuna in those years is the description dated 1593 that can be found in ‘Storia 
del Friuli’ by Germano de'Vecchi. 
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As already mentioned before, the descriptions and positions of each statue as reported in 
the two inventories are incomplete and often approximate and they lack a graphic appara- 
tus able to convey the exhibition in its original aspect with the correspondent disposition of 
the sculptures in the Tribuna. 

Among the various studies about the Tribuna of Palazzo Grimani and its collection of stat- 
ues, one that is worth to remember is that of the scholar Marilyn Perry [4] who, in 1972, was 
able to reconstruct the current disposition of the statues as reported in the inventories of 
Pellegrini and de’Vecchi inside the Archaeological Museum of Venice using also the drawings 
made by Anton Maria Zanetti ‘the Young’ in 1736. This contribution is a fundamental stage 
in the attempt to virtually reconstruct the original exhibition of the Tribuna. Unfortunately, 
Perry's study did not manage to reconstruct the exact disposition of the statues. A first and 
plausible hypothesis, then widely supported by critics, can be found instead in the study of 
the scholar Eva Soccal [5]. In her graduation thesis of 1999, Soccal hypothesised that the 
inventory of Pellegrini followed a circular trend in the enumeration of the statues, starting 
from the sculptures situated at the base of the walls and then rising helically upwards. The 
idea suggested by Soccal became an installation in an exhibition that took place in Bonn 
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in 2002 and that was dedicated to collectors and Venetian art. The images of the walls of 
the Tribuna were printed in a l:l scale and used as background for some of the original 
sculptures lent by the Archaeological Museum of Venice. Drawings by Zanetti were instead 
used as background for the sculptures that were supposedly originally placed in niches and 
on upper shelves. 

t is probably superfluous to point out that this research [6], mainly focused on the digital 
reconstruction of the Tribuna, had to rely upon reconstructions made by art historians and 
archaeologists who have received great credit from the scientific community. 

Apart from the already mentioned Soccal’s study, the other most important research is ‘La 
Tribuna ritrovata. Uno schizzo inedito di Federico Zuccari con l'Antiquario dell'illustrissimo Patriar- 
ca Grimani’ by Irene Favaretto and Marcella de Paoli. The work of the two Venetian scholars 
has been inspired by the finding of a drawing that apparently has nothing to do with the 
Tribuna of Palazzo Grimani. This drawing by Federico Zuccari is a reproduction of ‘The Feast 
in the House of Simon the Pharisee’ made by Paolo Veronese in 1573 [7] and is preserved in 
the Department of Prints and Drawings of the British Museum in London. On the back of 
this drawing there is a freehand sketch made by the author himself which represents an im- 
age, although partial, ofthe Tribuna of Palazzo Grimani when it was full of Greek and Roman 
statues. The sketch represents the right half of the wall facing the entrance of the Tribuna. 
Although the drawing is incomplete, Favaretto and de Paoli have tried to reconstruct, even 
though only on paper, the entire room, hypothesising its organisation in the 80s of the | 6th 
century with the repositioning of about 100 statues. (fig. |) 

Discussing the specifics of the digital reconstruction, after an intense study of the sources 
with the collaboration of art historians and archaeologists, the first operation carried out 
was a laser scan survey of the space of the Tribuna and the creation of a physical 3D model. 
Moreover, once the statues were positioned in their current location, some of them were 
surveyed using digital photogrammetry, in particular the Hora d’Autunno, which is nowadays 
preserved in the antitribune hall of Palazzo Grimani. They survey and modelling of all the 
statues was a long and complex work that involved only a few cases in this first phase, trying 
to build and explore a methodology that was then completed thanks to the collaboration 
with the Directorate of Cultural Activities and Sport of the Veneto Region, which lent the 
models of the missing statues. The data regarding the room and the statue obtained through 
the photogrammetrical survey were then processed through the software Agisoft Photoscan. 
The elaboration of the room was meant to obtain a three-dimensional model suitable to 
be then remodelled with a software of digital drawing in order to obtain a printable model. 
The elaboration of the statues was instead meant to obtain a texturised model that could 
be used to create the content of the application in augmented reality. The modelling made 
with the software Rhinoceros created a simplification of the digital clone of the room in terms 
of geometry and decorative details that are believed to be dated after 1582 — the reference 
year of the hypothetical reconstruction. In particular, the floral decorations at the centre of 
the ceiling coffers of the vault, the plaster masks placed above the round arches of the main 
niches and the plaster survey that adorned the mirrors above the arches were removed. In 
addition, the entrance on the left wall and the window on the right wall were removed for 
the same reasons and they were replaced with niches. 
To enable the view inside the room in the printed physical model, only one half of the room 
was printed — that is, the one resulting from the division of the room itself by means of a 
plane perpendicular to the floor and passing through the vertical axis of the left and the 
right walls. The half that was printed is the one that contains the wall facing the entrance of 
the Tribuna. The model was subsequently divided into independently printable sections that 
were then united among themselves with hooks (between the walls, and between the walls 
and the sections of the vault) made ad hoc or with a glue (between the sections of the vault, 
and these and the skylight). After the printing phase, an application in augmented reality was 
created using the Unity and Vuforia software. In particular, Unity is a software that enables 
to create a large number of applications such as apps for mobile devices, while Vuforia is a 
kit for the development of applications in augmented reality for mobile devices. The struc- 
ture of the application includes a simple main menu that introduces the augmented reality 
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scene and explains to the users how to use it through the ‘instructions’ command. That same 
application contains the interactive view of the original position of fourteen statues inside the 
Tribuna, that is, those relating to two thirds of the wall facing the entrance of the room. (fig. 2) 
The software Rhinoceros was initially used for the creation of the scene with the digital 
models of the statues to equalise the scale of the mesh models of the statues and that of 
the model of the room. Then, the models of the statues were virtually situated inside the 
Tribuna. Finally, the whole complex of the statues was exported in .obj format and import- 
ed into Unity to create the scene. Therefore, by framing the model of the Tribuna with a 
normal mobile device and thanks to some targets situated in the different support points 
of the statues, the image of the Tribuna with its statues appears. The next step consists of 
experimenting this application in the physical space of the Tribuna, considering the excellent 
functioning of the same on the scale model. The application designed in augmented reality 
offers interactive in-depth contents too: each statue has a multimedia card, consisting of a 
panel containing its real image, its representation made by Zanetti in his catalogue of | 736, 
and a small text with its name, its current location, and some other information. This panel 
can be activated for each statue by clicking on the statue itself. 


Notes 
[1] Cf: Moschini 1815. 
[2] Sansovino 1581, pp. 138-139. 


q 


Cf.: Inventario Pellegrini, 16 november 1593 (asv, Procuratori de Supra, b. 68, proc. 151, fasc. 3, |, cc. 33-42). 
[4] Perry 1972, pp. 75-253. 


[5] Soccal E. 1999-2000, La Tribuna di Palazzo Grimani. Ipotesi di ricostruzione di una raccolta d'antichità nella Venezia del XVI 


secolo, graduation thesis, University of Padua, supervisor Graduate |. Favaretto; Soccal 2002, pp. 447-455. 


[6] Part of this work has been developed in the thesis Nuove Tecnologie per un Exhibit Museale Innovativo, Graduate Valeria 
Sambucini, for the ll-level Master's degree Mi-Heritage Sistemi Interattivi e Digitali per la Restituzione e Valorizzazione del 
patrimonio Culturale of the luav University of Venice, scientific manager Prof. Giuseppe D'Acunto. 


[7] For Federico Zuccari, see: Acidini Luchinat 1998 e Acidini Luchinat 2001, pp. 235-240. 
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REPRESENTATION CHALLENGES 
Augmented Reality and Artificial Intelligence in 
Cultural Heritage and Innovative Design Domain 


The Rock Church of San Micidiario of the 
Pantalica Site and 3DLAB VR/AR-Project 


Giuseppe Di Gregorio 


Abstract 


This work gives the first results of a research project funded by the Sicily region, aimed at the creation 
of virtual reality (VR), augmented reality (AR) models, in the context of some country of the terri- 
tory of Sicily with UNESCO listed monuments. This project sees the collaboration between some 
universities on the island and some partners who produce software and automated robotics systems 
and Al. Below are the first results of the workflow produced for the creation of 3D models of one 
of the three rock churches of the necropolis of Pantalica: an Unesco site. It also describes the pro- 
cess underway to get to the creation of VR and AR models to be used with three different levels of 
immersive reality: normal commercial viewers, different types of Oculus viewers, or in special virtual 
rooms Cave Automatic Virtual Environment (CAVE), already existing or created within the project. 


Keywords 
3D survey, digital survey, SFM, photogrammetry, VR/AR. 
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Introduction 


The ever-increasing and increasingly cheaper availability of active and passive sensors, such 
as 3D laser scanners and sophisticated photogrammetry systems, with increasingly versatile 
and feature-rich software environments, on the one hand, and fixed equipment on the other 
with wearable devices for the use of highly immersive environments has made it possible to 
create a real market of Virtual Reality (VR) and Augmented Reality (AR) which, according to 
the Worldwide Semiannual Augmented and Virtual Reality Spending Guide by IDC, has ac- 
celerated over the last five years, with a compound annual growth rate of 198%, to the point 
of reaching $ 143.3 billion in 2018.VR applications are increasingly widespread and numerous 
in many fields, such as: medicine, biology, automotive , manufacturing, cultural heritage, public 
administration, “internet of things”, architecture, gaming, advertising, online sales, communica- 
tion and marketing, tourism, and publishing, fashion, education, training, etc. and the emerging 
integration ofVR into business processes such as design, production, simulation, maintenance, 
safety, etc. is extremely significant. It is therefore no wonder that advanced visualization VR/AR 
laboratories are in continuous creation and development, both in an academic and industrial 
environment, often with public-private co-management and that in 2017 the European Com- 
mission funded the VISIONAIR project, which had the goal, successfully achieved, to create 
a continental network with more than 20VR and 3D visualization centers and to validate it 
with about 200 applications and projects of cultural heritage, biology, chemistry, engineering, 
mechatronics, medicine, and much more, selected through competitive procedures. 


The 3DLAB Sicily Project 


As part of a project funded by the Sicily region, the Department of Civil Engineering and 
Architecture of the University of Catania (DICAR) participates in the development of VR 
and AR models, through a research group related to the disciplines of design and represen- 
tation, which intervenes for some Cultural Heritage, in the context of various municipalities 
already registered as UNESCO sites. Among these it is worth mentioning the country of 
Sortino (SR) with the important and vast archaeological site of the necropolis of Pantalica. 
Among the various purposes: — improve and enhance the use and enhancement of the 
artistic, cultural and environmental resources of the territory, given that virtual reality envi- 
ronments are becoming one of the impulses of tourism 4.0., — use technologies to create 
applications that allow an extremely dynamic and multisensory interaction of the island's 
cultural heritage, — create, develop, validate and promote a sustainable regional network 
of some centers for VR/AR and 3D visualization, — federate the infrastructure with that of 
VISIONAIR, in order to create visibility at an international level; — create a service that bor- 
rows the concept of “liquid lab” or “liquid studio”. 


Fig. |. Pantalica (SR), 
exterior of the rupestrian 
church of San Micidiario. 
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Fig. 2. Pantalica (SR), the 
area adjacent to the 
oratory of San Micidiario. 


The Rock Churches of Pantalica 


Declared a UNESCO site in 2005, the site of Pantalica consists of the Anaktoron or prince's 
palace dating back to the protohistoric period, the remains of an inhabited area of the same 
period, about 5,000 tombs in grotticella and the remains of three Byzantine villages linked 
to three rock churches also called oratories. The entire site is marked by the path of two 
rivers, the Calcinara and the Anapo and consists of different areas defined with different 
toponyms. The three rock churches are those of San Micidiario, of the Crocifisso and of 
San Nicolicchio, located on three different sides, each serving a surrounding village. Paolo 
Orsi identified for these oratories a high medieval rock settlement, expression of a natural 
Byzantine fortress. Furthermore, the same archaeologist placed the chronology of the three 
churches prior to 878 [1]. Aldo Messina in updating the catalog of the rupestrian churches 
of Syracuse, corrected the dating by proposing to place them in the context of the Norman 
period. The first rock church on which attention was paid within the 3DLAB project was that 
of San Micidiario (fig. |), located on the edge of a large rock village located near the saddle of 
Filiporto: a fortified trench. Three distinct rooms (fig. 2), communicating with each other are 
part of this oratory, of which only the first two with an access from the outside. The first room 
is the one intended for worship and consists of a hall and a presbytery, the hall has a regular 
shape of about 4.00 x 2.70 m, while the presbytery has dimensions of about m 2.70 x 2.00 m, 
while the presbytery has dimensions of about m 2.70 x 2.00 and occurs at a greater share of 
the room that precedes it, the other two rooms with an almost regular plan, have dimensions 
of about m 3.90 x 3.80 and 2.65 x 4.24 m. In the cult environment, in the area that separates 
the two zone, the attacks of the rock of a pre-existing templon are still visible, which would 
strengthen the Byzantine origin with three openings that repeated the geometry of the apse 
area. Orsi tells us that he found a painted plaster in the apse area, of which only traces are vis- 
ible today. The information that a poly-figurative composition existed in the apse area remains 
confirmed, the issue was dealt with and deepened by G.Arcidiacono [2]. 


The Creation of the 3D Model and VR/AR 


The first phase of processing involved the use of SfM, the equipment used is a Canon full 
frame digital camera. To create the 3D model, the multi-image photogrammetry application 
SfM Zephyr by 3DFLOW was used (fig. 3). The result was a unique 3D model for the 
three environments, with the ability to navigate continuously within the three volumes. After 
checking other aspects, several elaborations followed, including the section with a horizontal 
plane to obtain a section in 3D. This acquisition and processing through multi-image pho- 
togrammetry is accompanied by a scan with the Faro 350 plus Laser Scanner, in order to 
evaluate the results for the definition quality of the rock walls, for the creation of the VR 
model between the two systems. The 3Dlab project involves the use of three VR CAVES. 
The first VR CAVE already operational, was developed by SWINGIIT [3], and is used for 
production of applications, as part of research and development projects and for services to 
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Fig. 3. Pantalica 

(SR), oratory of 

San Micidiario, the 
presbytery zone, Zephyr 
reconstruction. 


Fig. 4. SWING:IT VR 
CAVE video projectors. 


Arduino 


Gee re of the Utente Applicazione 
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third-party activities that require an advanced visualization infrastructure. The Virtual Room 
is a structure capable of reproducing an immersive virtual reality experience IVR (Immersive 
Virtual Reality), organized on a video-theater inside which images are projected using 3D 
projectors (fig.4) and/or screens, together with appropriate 3D software. One of the strengths 
of the SWING:IT system is the presence of the tracking sub-system, which allows you to 
track the user's position, using this information for real-time recalculation of the perspective 
vision, thus giving the user the sensation of navigation within the 3D scene. The SWING: IT 
virtual room is controlled by two PC, the rendering node (Render PC), dedicated exclusively 
to scene rendering and the processing node (Master PC), which takes care of the software 
execution.Video projectors take care of projecting 3D images on the panels, with a projector 
for each panel. The tracking system adopted is the ART TRACKS, a stand-alone system, which 
uses an infrared (IR) tracking method, equipped with a central unit that holds the tracking logic 
to which the infrared cameras refer Finally, a “Pattern Recognition” system is able to determine 
the position and orientation of each individual marker present within the tracking area, using 
the Hand Track and Crystal Eyes as input peripherals. Therefore, additional input devices are 
being evaluated to be integrated and tested to improve the immersive experience within the 
Virtual Room, while avoiding introducing additional disturbing elements to users. As part of 
the development of the project 3DLAB, SWINGIT is evaluating the possibility of expanding 
the list of input devices in order to improve the immersive experience (fig. 5). The 3D scenes 
are created through the Unity 3D development platform, the application generated produces 
a video output that is managed by the TechViz XL software, which is able to view the 3D 


+ 


scenes without any limitation of resolution and performance. 


Acknowledgements 

This work, equipment illustrated were supported by the project “Creazione di una rete regionale per l'erogazione di servizi 
innovativi basati su tecnologie avanzate di visualizzazione” (3DLab-Sicilia), Grant No. 08CT4669990220, funded by Opera- 
tional Program 2014-2020 of the European Regional Development Fund (ERDF) of the Sicilian Region. 

Notes 

[I] Arcdiacono G., 2019, Le ultime fasi di Pantalica: le chiese rupestri e la loro decorazione pittorica, sta in Atti del convegno 
di Sortino (SR) 15-16 dicembre 2017, Consorzio Universitario Archimede Soprintendenza BB.CC.AA. — Siracusa, Scuola di 
specializzazione in Beni Archeologici — UNICT, Comune di Sortino, Bottega D'Erasmo Aldo Ausilio Editore in Padova, p.204. 
[2] Arcidiacono G., 2019, pp. 203-221 

[3] SWING IT is the acronym or abbreviation of Software Engineering based in San Giovanni La Punta (CT), and is one ofthe 
partners of the 3DLAB SICILY project. 
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REPRESENTATION CHALLENGES 
Augmented Reality and Artificial Intelligence in 
Cultural Heritage and Innovative Design Domain 


Understanding to Enhance, Between the 
Technical and Humanist Approaches 


Elena Ippoliti 


Abstract 


The topics brought to attention by the Symposium provide the opportunity to reflect on research 
experiences in the last fifteen years where the potential of digital technology, and augmented reality 
in particular, has been tested to enhance the cultural heritage. It is an opportunity to verify — beyond 
not only the apparent kaleidoscope of the latest technological “novelty or wonder”, applications, and 
goals, but also changing groups of scholars or case studies — the presence or lack of a driving motive 
with regard to both the general goals and the specific discipline of Disegno, thereby validating, again 
at a distance of several years, the different experiences. 


Keywords 
representation and communication of cultural heritage, representation of the city, communication 
and visual perception, gamification, storytelling. 


COMUNICAZIONE [MN 
BENI CULTURALI 


Fig. |. The figure wants to 
summarize the common 
purpose of a set of 
researches: publishing 
and sharing bodies 

of knowledge about 

the cultural heritage 
organized through 3D 
models integrated with 
the real, geographical, 
and web spaces. From 
Maps, technological 

and spatial models for 
understanding, promotion 


and sharing urban heritage, 


PRIN 2007-2009” and 
Towards the construction 
of a Digital Atlas for the 
documentation of cloisters 
and courtyards in Ascoli 
Piceno, 2006-2008. 


The Model 


The starting point of the reflection lies in research [|] that has selected critical reflection 
and experimental investigation of digital technologies as a favoured area to enhance the 
cultural heritage by defining “visual models for knowledge and use” [2]. 

The first approach was mostly aimed at faithful, imitative modelling of reality with a focus on 
solving problems related to the details of reconstruction - number of polygons, rendering 
quality, textures, lighting, etc. This setting conditioned the possibilities for online exploration, 
preventing correspondence with the goals of the research, i.e. publishing and sharing bodies 
of knowledge about the cultural heritage organized through 3D models integrated with the 
real, geographical, and web spaces. 

Technological developments (from miniaturization of the components to identification of 
algorithms for data compression) already allowed for some hybrids between real and vir 
tual, variously refined by the subject's level of interaction with the real space, presence, and 
type of device. In particular, the first applications of augmented reality — or more properly 
mixed reality — had already been developed for smartphones (having just been equipped 
with GPS, electronic compass and inclinometer, camera, and wireless connection), although 
severely limited in their integration between real/natural and virtual vision, frequency with 
which the virtual scene was updated, adherence of this to the real space, etc. 


Tecnologie _ WEB 
: ies DBMS (web gis) 
i * GRASS + PostgreSQL + MapServer 
ž *QGIS + Postgis +2,5dMap 


Nevertheless, pursuing the objectives of the research and relying on technical/procedural 
expedients to overcome the technological limits, we explored the entire arc of the real-vir- 
tual continuum, trying to variously combine the different terms in play (real, virtual, devices, 
etc.) to probe the different perceptual responses. 

In some way, we practised the three-dimensional taxonomy developed by Milgram and 
others [Milgram et al. 1994; Milgram et al. 1995] on AR applications obtained from the 
entire mixed reality spectrum. In this taxonomy, two dimensions (reproduction fidelity and 
extent of presence metaphor) are closely related because the question of realism (or better 
yet, plausibility) of the scene is related to the measure with which subjects perceive their 
participation in the scene — from either outside or inside. The third dimension (extent of 
world knowledge) instead measures the subject's judgement of the integration of the virtual 
content with the real world. This, however, is not only a function of the viewer's exact posi- 
tion in the scene, but also where the intelligence of human perception intervenes to "close 
the cycle” [Milgram et al. 1995, p. 287]. 

In this framework, the “visual models of knowledge and use” were modified — from the “3D 
digital representation/model” to the “informed 3D model” to the “digital 3D scene” [Ippoliti, 
Meschini 2010] — with an ontological shift in the representation/vision from ‘objective’ to 
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Fig. 2. AR experiences 
about Piazza Arringo, 
Ascoli Piceno. From 
Informative integrated 
Models to know, improve 
and share urban and 
environmental heritage. 
Testing 3D interfaces for 
“cultural and geographic 
objects’: the architecture 
of the information and 


computerized architecture”, 


PRIN 2009-2012” 
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‘subjective’, turning the observer into a spectator and then an actor, no longer in front of the 
representation but within the scene. 

The ‘digital 3D scene’ became a participatory place by virtue of which one could effectively realize 
the construction, access to, and sharing of cultural content. Different types of ‘visual 3D models’ 
have been explored in this view for a similar number of applications and tours relying on aug- 
mented reality and virtual reality according to different degrees of interaction and/or immersion. 


The Topic 


In this framework, we tested different applications of AR (or, as mentioned above, MR) on 
very different case studies (a neighbourhood, square, set of goods with uniform characteris- 
tics, etc., but also a collection of representations of an urban space, square, etc.), all of which, 
however, were types of cultural goods that transmit their value starting from their state as a 
‘figure’. These goods should be used consistently with their specific signs in visual languages 
that encourage modes of exploration, thus consistent with the goals of the experimentation: 
interpreting the simulation of the space and the set of visual technologies for accessible, 
participatory, and involving communication of the heritage. The “3D scene” is therefore the 
key to access the cultural heritage because the emotional emphasis tied to vision activates 
involvement, encouraging participation and turning the cultural good into an experience. 


The case studies, however, are always anchored to the authentic motive for the research, which 
originates in defining the cultural good itself as the expression of the system, that is, the set of 
qualitative and quantitative connections between the individual goods and between these and 
the context. This is a fundamental awareness for Italy leading to the culture of enhancement, 
“in which the value of each individual monument or object of art results not from its isolation, 
but from its insertion in a vital context” [Settis 2002, p. 15]. The process of building knowledge 
therefore cannot be limited to the good in itself, but should express its interactions — both 
physical and linguistic — with the context where it originated, making the fabric of relationships 
that give it substance and clarify it explicit and tangible. It is in this context that we should begin 


+ 


to explore and learn about the “evidence with civic value”. 


The Map 


Indispensable in this revisitation is another body of research focused on ‘maps’ [3], a logical 
paradigm and technically concrete place in which information is always contextual. Each datum 
is associated with a specific location in the map space, and it is this space itself that allows the 
nodes between data and positions to be highlighted. Maps are an organized form of anthropic 
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Fig. 3. AR experiences 
about the Archaeological 
Museum, Palazzo Panichi, 
Piazza Arringo, Ascoli 
Piceno. From top to 
bottom: AR mobile 
device: simulations of a 
virtual reconstruction 

of the ionic column; AR 
desktop application: the 
sepulchral Epigraph of 
the concubine Pontia 
Callista. From Informative 
integrated Models to know, 
improve and share urban 


and environmental heritage. 


Testing 3D interfaces for 
‘cultural and geographic 
objects’: the architecture 
of the information and 
computerized architecture, 
PRIN 2009-2012”. 


space that enable the complexity of reality to be penetrated through reduction into a model, 
but it is above all a means in which the individual parts and whole are understood through 
relationships, the only thing that can define the contextual meaning of the phenomenon. 
These reflections led us first to experiment with the different means through which a map/ 
representation communicates: icons, through similarity with the object, in which the infor- 
mation/communicational flow is continuous; and language, through signs and symbols, in 
which the flow is discontinuous. 

These then led us to test different enunciative means of cartographic ‘discourse’: description 
and narration [Marin 2001]. In description, the gaze is panoptic and simultaneously embraces 
multiple points of view, expressing an atemporal spatiality. In narration, in contrast, the view is 
that of a traveller crossing spaces and itineraries and the points along the path follow a spatiality, 
expressing the temporal dimension. 

The sense of the AR and MR experiences falls within these reflections. They are methods and 
devices to create visual models designed as interfaces to share content between the transmitter 
and subjects, for whom viewing becomes an experience, encouraged as they are to participate 
in the message of communication. The range of infographic methods only acquires a sense when 
viewed in light of the application of contextualizing the information, that is, the system. 


] COLONNA IONICA 
y 


pee 


pa FFIGRAFI 


According to this visual, the common thread crossing and linking earlier applications and 
many more recent ones is very clear: the conviction that representing the inhabited space is 
still a necessity. This conviction expresses the primary goal of the research, i.e. reinterpreting 
urban representations, thus taking advantage of all the opportunities offered by visual tech- 
nologies. lt means revisiting the ‘representation of the city’ because due to the emotional re- 
lationship historically tying communities to places, it may still today be a vehicle of emotions 
for suggesting histories and interpretations, an interface for initiation to knowledge about 
values and deeper meanings of the city and cultural heritage, and therefore an indispensable 
tool for its enhancement. 


Conclusion 


This revisitation has shown that the role of technological innovation, and AR in particular has 
always referred to the goals of the research: collaborating to build knowledge that can be 
used and enjoyed by a broad public, testing perceptual interfaces that lead to an increase in 
levels of the subject's interaction with the cultural good. The research therefore features the 
general goals of identifying ‘visual models of knowledge and use’ of the goods that are clearly 
based on rigorous means, but also amplifying the representative sense, using the most useful 
technologies to do so. 
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More in general, different research projects have used AR (recently integrated with the 
techniques of storytelling, gamification, and storydoing) [4] to transform knowledge about 
the cultural good into an act of enhancement, and therefore the "visual models of knowl- 
edge and use” into an ‘experience’ not only of the individual object, but of the whole, that 
is, the system of relationships. This general goal is the starting point for designing paths of 
historically and culturally consistent meaning to explore “evidence with civic value”. 

The applications of and motivations for the experiences are all framed within a single hori- 
zon that interprets the specific discipline of Disegno in which the role of informational da- 
tum is continuously exchanged with the role of the image, which not only represents it, but 
embodies it in tts essence [Cervellini, Ippoliti 2005, p. 75]. By virtue of its spatial/topological 
connotation, the image renders information by giving it a form at the intersection of three 
themed objects: the model (iconic), map (also a type of model), and topic (in the enunciative 
meaning still a model). This common thread — very long, invisible, and indestructible — “can be 
disentangled without undoing everything’ and in which “even the smallest fragment can be 
recognized” because it pertains to the system [Goethe 201 |, p. 187]. 


Notes 


[I] For reasons of space, the research referred to in this article cannot be detailed. See, however, the notes, references, 
and figures. Many researchers have participated in different experiments, including Francesco Cervellini, Alessandra Meschini, 
Daniele Rossi, Mariateresa Cusanno, Annika Moscati, Jonathan Sileoni, and Danilo Spinozzi at the University of Camerino, and 
Andrea Casale and Michele Calvano, Cristian Farinella, Lorena Greco and Stefano Volante at the Sapienza University of Rome. 


[2] Among these research projects, the primary ones include Maps, technological and spatial models for understanding, promotion 
and sharing urban heritage, PRIN 2007-2009, coordinator Mario Centofanti, University of Camerino research unit leader Elena 
ppoliti; University of Camerino project Towards the construction of a Digital Atlas for the documentation of cloisters and courtyards 
in Ascoli Piceno, 2006-2008, principal investigator Elena Ippoliti. 


[3] The primary research includes Informative integrated Models to know, improve and share urban and environmental heritage. 
Testing 3D interfaces for ‘cultural and geographic objects’: the architecture of the information and computerized architecture, PRIN 
2009-2012, coordinator Mario Centofanti, Sapienza University of Rome research unit leader Elena Ippoliti. 


[4] These research projects include the Sapienza University of Rome project Between museums and cities: ‘cultural heritage at play’ 
between edutainment and gamification. The role of representation between a technical and humanistic approach, 2020-202 |, principal 
investigator Elena Ippoliti; Usage/knowledge systems in museum communication, 2018-2019, principal investigator Andrea Casale; 
Representations of cities and cultural identity. New guides between digital technologies and visual itineraries for the enhancement of the 
city’s cultural heritage and tourism, principal investigator Elena Ippoliti. 
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Illusory Scene and Immersive Space in 
Tintoretto’s Theatre 
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Massimiliano Ciammaichella 


Abstract 


The perspective analysis of some canvases by Jacopo Robusti is the result of a research which involved, 
within the University of Padua, architecture and art historians with representation scholars, to share knowl- 


edges about one of the most important Venetian art 


tists of the second half sixteenth century. The study, 


continued in collaboration with the luav University of Venice professors and students, focused on an exhi- 


bition proposal at the Sala Capitolare of the Scuola Grande di San Marco in Venice, where hopefully some 


of paintings will return to their original destination. 


The main objective concerns the creation of an experiential itinerary organized by a sensitive narration 
that starts in the same act of its fruition, to allow the observer a critical interaction dependent on knowl- 
edge process of the works and their evoked spaces, without the aid of controllers or digital viewers. 


Keywords 


Tintoretto, architectural perspective, exhibit, video mapping, scenography. 
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The essay documents the results of a research aimed at enhancing the artistic and cultural 
heritage offered by Tintoretto’s work, through an exhibition mode able to guarantee a 
knowledge experience increased using advanced digital interaction devices. 

Starting from the most up-to-date artistic historiography of Tintoretto's work, the analysis 
conducted highlights the Venetian artist's relationship with architecture, which has remained 
unexplored or limited to a few essays that underline an uncommon spatial research. Con- 
siderations of the scholars Sergio Marinelli, Erasmus Weddigen, Michael Matile and Martina 
Frank have laid the basis for a further in-depth investigation of the relationship established 
by Tintoretto with the painted space, based not only on a precise knowledge of built archi- 
tecture, but also on a profound admiration for the theatrical scenic space [Marinelli 1980, 
Weddigen 1991, Matile 1996, Frank 1996]. 
Following the renovatio urbis, imposed by the Doge Andrea Gritti — for which the recon- 
figuration action of Venice image is evident —, the theoretical and material examples due 
especially to Sebastiano Serlio and Jacopo Sansovino, together with the uncontested con- 


+ 


tact with Tuscan-Roman figurative culture, become a source of inspiration and pictorial 


+ 


transposition of the urban environment proposed by Tintoretto. 
In his early paintings the architecture interest took on a secondary role, relegating it to the 
background of the narration, but from the mid—Forties of the sixteenth century, with the 
Christ among the Doctors, Christ and the Adulteress and Miracle of the Slave, the architec- 
tural space component became an integral part ofthe composition, which focuses on linear 
perspective. This method is understood as a technical and conceptual means of controlling 
the spatial configuration, alded using papier-máché maquettes in which the painter, accord- 
ing to Carlo Ridolfi— who met Tintoretto's son, Domenico — placed wax and clay statuettes 
to verify the position of the characters dressed in rags, on which he carefully elaborated 
folds and poses of the limbs. In other cases, if necessary, he increases their scale and sus- 
pends them with wires from the beams to check their foreshortening, then he builds "small 
houses and relief-perspectives made of boards and cardboard, inserting small lights for the 
windows” [Ridolfi 1648, p. 7] and uses candles to simulate the different light sources to be 
reproduced in the painting [Marinelli 1980, p. 319]. 

This research, found in the scenic practice, can be traced back both to his theoretical edu- 
cation, which took place in Sebastiano Serlio's books in which we find descriptions of the 
performant spatiality and indications on the arrangement of lights, and to the painter's famil- 
iarity with theatrical spaces and, in particular, his friendship and frequentation of Ruzzante, 
Pietro Aretino and Andrea Calmo [De Vecchi 1972, pp. 101-132]. 

t is evident that the influence exerted by Serlian representations of ephemeral theatrical 
scenes — combined with the architectural practice and perspective skills of the Bolognese 
architect — adheres to Sansovinian petrified translation of Doge Gritti's idea, leading Tin- 
toretto to transform the painted space into a credible urban stage on which to place the 
biblical narrations. 


Research and Case Study 


The research establishes the aim of analysing the coherence, verifying the precision of per- 
spective technique used, also reflecting on the intentional deformations and pictorial cor 
rections chooses by Tintoretto, in a strategic and controlled approach, especially in his more 
mature works. A predominant architectural component is evident in these words, and so 
we have tried to understand: the relationship with the biblical events narrated; the scenic 
setting in which the painted architecture becomes the structuring foundation of the entire 
composition; the visual and physical kinematic, regulated by the point of view of the ob- 
server forced to move in the multiple narration of the mise-en-scene. The implementation 
of precise perspective rules offered the possibility of restitution the painted space, with a 
rigorous inverse method and with adequate mathematical modelling software, demonstrat- 
ing, in enhancing the written historical evidence, the real possibility that Tintoretto used real 


small-scale maquettes, necessary to compositions design in which to insert the lights and 
then the characters that inhabit them (figs. 1-2). 
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Fig. |. The Miracles of 
Saint Mark, perspective 
restitution [Gabriella Liva 
2020]. 


Fig. 2. Architecture 
reconstruction and 
perspective configuration 
[Gabriella Liva 2020]. 


The canvas dedicated to The Miracles of Saint Mark is part of the chosen case studies and 
represents a masterfully constructed setting, made even more credible when associated, in 
Weddigen's hypothesis, with two other masterpieces: the Removal of the Body of St Mark 
and St Mark Saving a Saracen During a Shipwreck, in its primordial composition [Grosso 
Guidarelli 2019, p. 106]. If it is considered singularly, or in a unitary perspective strategy, at 
the moment of its appearance inside the Sala Capitolare of the Scuola Grande di San Marco, 
it aroused particular astonisament, if not the wonder ensured by the attempt to merge the 
hall real space with the place of the depicted scene. The context is represented by a majestic 
old-style portico that rises above a polychrome chessboard floor, making the articulated 
composition not immediately intelligible. 


As with other works, here we are not dealing with a static setting relegated to a simple back- 
ground, but with a geometrically structured space in a direct dialogue with the characters who 
activate it, participating in their actions or accompanying the viewer in specific painting areas, 
so as to encourage a multiple reading of the events narrated. The study of the work included 
a detailed analysis of the various compositional elements that make it up, the identification 
and classification of which become of primary importance for the exhibition layout proposal. 
Inside the space, punctuated by the rhythm of the cross vaults interspersed with large round 
arches, “there appear in a long portico many sepulchres hanging on the walls, drawn by a 
beautiful perspective” [Ridolfi 1648, pp. 14-15], whose orthogonal profiles to the painting 
converge towards a single vanishing point. It is positioned on the left side in correspondence 
of the pulse of the saint who indicates, to the viewer the recognition of his body. The charac- 
ters depicted are distributed along the nave, according to three registers: in the foreground 
a group of people is represented on the right, including a possessed man taken from behind 
and breathing sulphurous vapours; in the centre the client Tommaso Rangone kneeling and, 
on the left, the majestic figure of St. Mark in the act of revealing the discovery of his earthly 
body; then the viewer assists the extraction of the body from the hanging sepulchre and, in 
the background, the vain search for the body of the saint inside an earthly sepulchre. 
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Fig. 3. Exhibition setting 
proposal, Sala Capitolare 
of the Scuola Grande 

di San Marco, Venice 
(Thomas William 
Propedo, 2020). 


Fig. 4. Visualizing 
Tintoretto film frames 
(Thomas William 
Propedo, 2020). 


Methodology 


Considering the narrative sequence of painted events, the study, continued in collaboration 
with University luav of Venice professors and graduating students, focused on an exhibition 
setting proposal and visual entertainment to be reproduce in the Sala Capitolare of the 
Scuola Grande di San Marco in Venice, trying to retrieve the close relationship between the 
real space of the hall and the illusory space evoked by represented scenes (fig. 3). 

The main objective concerned the creation of an experiential itinerary, organized by a 
sensitive narration that starts in the own act of its fruition, such as to allow the observer a 
critical interaction depending on the knowledge process of the works and the geometries 
evoked by their spaces. The installation includes the projection of multimedia contents, to 
be displayed on semi-transparent screens, and it can narrate the compositional structure of 
the works in a controlled deconstruction of the significant painted elements. 

Protagonist is the video mapping: it marks the times and dynamics of a possible and credible 
mise-en-scene, inside an autonomous stage, hosting different multimedia systems necessary 
to the experience. The visitor is led by a light path and voice of an actor in catch a show 
that opens its perspective—linear registers, hastening the visualization of material physicality 
of the environments, in which the protagonists of the works are animated by geometric 
transformations typical of image warping. 
According to this exhibition logic, this following proposes an immersive fruition path, aimed 


+ 


to enhancing the interaction and knowledge of the inestimable artistic and cultural heritage 
offered by Tintoretto's works, in a close to theatrical dynamics approach [Propedo 2020]. A 
stage, extended along the short side ofthe room and about 26 meters long, rises from the 


floor to a height of |.4 meters to allow viewers to frame the three works showed and po- 
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sitioned in their original location, so as to enjoy them from their correct points of view. But 
this happens only at the conclusion of a specific narrative experience that anticipates the 
digital act of knowledge, mediated by a filter on which animated sequences are projected. It 
is possible to enter in the dark room guided by the cone of light emitted by a spotlight that 
invites the viewers group to place themselves in the correct positions, each equipped with 
people counter sensors that have the purpose of monitoring the access flow and activating 
the video projections on motorized semi-transparent screens, placed parallel and at a very 
short distance from the canvases. 


Conclusions 


The experience is combined by the voice of an actor, involved in three performing acts aimed 
at the decomposition and reconstruction of the peculiar characteristics of the works: the first 
one recalls the historical context and the significant steps that determined its realization, the 
second one describes the scene and the character roles involved in it, the third one analyses 
the pictorial, perspective and light techniques used by the artist (fig. 4). 

The scene space questions the value of virtual and augmented reality devices, commonly un- 
derstood as interaction filters with real artefacts, to enhance the learning act about the works 
which, in this case, are only revealed in all their tangible originality at the end of the experience. 
The expectation activated by the entertainment, planned within an multimedia guided 
scene, is confronted with the astonishment of the direct works observation, inviting to re- 
flect on their skiagraphy knowledge, just when the lights turn on and every filter disappears 
to reveal and paying tribute to the inestimable originality of Tintoretto's tangible works. Even 
if the place appointed to host them, ‘the same for which they were created and which de- 
clares their unitary conception, preserved from generation to generation, and so to speak 
from brush to brush, therefore has become over time less important than the processes of 
musealisation and decontextualization of works of art” [Settis 2017, p. 37]. 
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REPRESENTATION CHALLENGES 
Augmented Reality and Artificial Intelligence in 
Cultural Heritage and Innovative Design Domain 


Medma Touch, Feel, Think: Survey, 
Catalog and Sensory Limitations 


Franco Prampolini 
Dina Porpiglia 
Antonio Gambino 


Abstract 


The project is named Medma Touch, Feel, Think — Technological retrofit of the Archaeological Museum 
of Medma-Rosarno for the 3D catalog of the exibits works on display and the possible use by subjects 
with sensory limits’. We have carried out the 3D survey of all the main finds with analytical photo— 
modeling techniques, their scientific cataloging on ICCD's specifications, the creation of a website with a 
high interactivity content and an Application that allows the sharing of extended informations for blind 
people, through the combined use of analog 3D models and AR authoring software. 

A sustainable initiative, a driver for the technology transfer of innovation (often invoked, but rarely 
implemented), capable of generating ‘social empowerment’. The methodology can also be shared by 
small entities, but characterized by contents of high historical and cultural values, expecially if they are 
able to build-up a territorial network of high identitarian values. 


Keywords 
photo—modeling, scientific cataloging, inclusion, sustainability. 
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Fig. |. The Medma— 
Rosarno Archaeological 
Museum in Rosarno 
(RC). 

a. The large hall shaped 
ike an ideal “Via Sacra" 
and, in the background, 
he s.c. “Arula Tyro”. 
b.The large feminine 
erracotta busts 
showcase, from 

he sacred area of 
Calderazzo. 

c. The “Necropolis” hall: 
“pano view” from the 
Virtual Tour materials. 
(Ph. Gianluca Milasi). 
Note the “LOGES” tactile 
paving path set up by the 
project in the museum. 
d.The s.c. “town hall”, 
with the findings from 
the Gangemi collection. 


The Project rised from a proposal of the of Rotary International District 2100 (which also 
funded it extensively) and collects a broad institutional and social partnership: The ABAP 
Superintendence of Reggio C. and Vibo Valentia (a special thanks goes to Fabrizio Sudano, 
pro tempore manager of the Museum), the Italian Union of the Blind and Visually Impaired 
(UICI), The City of Rosarno, The Metropolitan City of Reggio Calabria, Terna SPA (which has 
effectively contributed to the financing) and many other Bodies and Associations that have 
contributed in different ways over time [1][2]. 

The Archaeological Museum of Medma-Rosarno is located in the archaeological park, in the 
town of Rosarno, full of olive trees that define its historical image in close connection with the 
actual perception of the territories, much like as they have been described directly by Paolo 
Orsi [Orsi 1913, passim] at the beginning of the last century on the occasion of the first major 
excavation campaigns. The exhibition is divided into three main sections. 

It starts with the reconstruction ofthe necropolis:the different types oftombs are reconstructed, 
while 10 small showcases — intentionally shaped in such a way as to recall the cemetery “nich- 
es'— contain a small but significant sampling of the sepulchral equipment. It then continues with 
splendid specimens of medmean coroplastic findings —statuettes of different sizes and shapes-, 
busts, large masks, cryophores, vases, and iron weapons found in the sacred area of Calderazzo, 
presented on the sides of a virtual ‘Via Sacra’. The last room contains the materials from the 
town, including a ritual fountain in terracotta and objects from the Giovanni Gangemi private 
collection, donated to the State, which consists of valuable vases with both black and red figures, 
including an amphora with scenes from the struggle for Achilles’ arms. 

The project can be summarized in five fundamental keywords. 


Surveying and Cataloging 


Over one hundred and fifty exhibits were surveyed with analytical photo-modeling tech- 
niques, reconstructed, and scientifically cataloged in a digital environment. The results, in terms 
of geometric precision and chromatic accuracy, are very satisfactory. The scaling and geometric 
verification of the models are carried out through the use of a calibrated tablet which contains 
30 markers of known coordinates singularly distinguishable by |2—bit encoding (fig. 2). The 
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Fig. 2. Votive altar, (aka 
“Ara della Fenice") upon 
the calibrated grid. 


Fig. 3. Catalogue form on 
CCD specification. 


Fig. 4. Ortoprojection and 
perspective section of a 
red figure vase. (invent. 
141437). 


Fig. 5. Full color 3D 
print reproduction of a 
medmean feminine bust. 


Fig. 6. 3D model (front) 
and a study of the actual 
deviations between the 
original point and the final 
mesh. 

“Overall surveillance” by 
a 5th century b.C. small 
Satyr bronze. 


markers have different sizes, to be significant regardless of the shooting distance and to en- 
sure rigorous verification of final residual errors. The precision obtained is firmly attested in a 
sub-millimeter range which makes the models themselves coherent with the l:1 scale survey. 
From the chromatic point of view —a critical factor in consideration of the particular nature 
of the relieved objects— excellent results were obtained thanks to a controlled shooting en- 
vironment and post-production of the photos, that included the ‘masking’ of non-essential 
elements for the restitution. 
The use of high-resolution cameras leads to very accurate modeling, which permits to push 
the analysis of single the objects to a very high level of detail, significantly expanding the 
possible critical data collection in fractions of time, if compared with any direct analysis, and 
at a very higher level of security (fig. 6). 

In many occasions it has been possible to extend the survey to the entire object surface 
(internal/external), making it the complete 3D analysis of the object immediately available, 
including the direct measurement of significant points of interest straight from the model, 
the automatic extraction of profiles, etc. (fig. 4). 

After a pre-cataloging phase based on the use of QR-Code and historical inventory numbers, an 
actual catalog form has been compiled, compliant with the ICCD specifications (RA-3.00) which 
permit direct compatibility with national databases on cultural heritage (fig. 3). 


3D reproduction 


Three-dimensional reproductions were made on one side to, somehow, replace certain im- 
portant finds transferred to other museums and, partly, to be used in an interactive application 
for people with sensory disabilities, both through direct full-color 3D printing in real scale and 
through a more ‘traditional’ procedure of sculpting reproduction controlled by the digital model, 
used for larger objects that were “oversize” for the 3D printing available technology (fig. 6). 
The final results, in terms of geometric precision and color accuracy, are very satisfying. 


Sharing and WEB 


A website has been created in a standard environment (WordPress) and permits to retrieve 
information about the museum, consult the multimedia catalog, and explore (or download, sub- 
ject to proper authentication) the three-dimensional models at different resolution. Full-res. 
models in 3D/.OBJ format are available (millions of vertices point clouds, 100 Mb or more), but 
we are working on smaller size models (around | Mb) in the GLB format by resampling the 
textures to grant sharability over smartphone even at a very low band connection. We are also 
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Fig. 7. Medma touch, feel, 
think App. From RAI 
TG-R reportage. Special 
thanks to RAI journalist 
Giulia Bondi and to 
Marika Meduri, president 
of unsighted association 
of Reggio Calabria (app. 
tester). 


ARTE PER NON VEDENTI 


formalizing a partnership with the IIIF consortium to reach complete control over the distribution 
of patented 3D models. The site also provides a high level of multidisciplinary interactivity allowing 
specialists and scholars, through Wiki-type pages, to collaborate with the implementation of the 
descriptive part of the forms. Finally, it contains a complete virtual tour of the museum, which is 
currently being programmed to allow a direct visual consultation of the catalog. 


The Touch, Feel Think App 


The core commitment of the project was addressed through the implementation of an 
application for unsighted persons, based on the use of high-resolution webcam and motion 
detection techniques that allow a completely hands-free approach. 

It starts with the 3D models (fig. |, 7a, 7b, 7c). People approach the replicated object and 
touch it without any restrictions: the system recognizes the touch and plays a first general 
soundtrack which introduces the historical framing of the object, and afterward, if the explo- 
ration continues, gradually recognizes the parts that are progressively touched: the description 
can then continue in detail, giving an account of many details that can create curiosity and 
encourage a scientific deepening of the knowledge of the assets, as well as their contextual 
conditions. The whole process is developed in an authoring environment, by which the signals 
coming from the cameras through the definition of any number of sensitive areas (hot-spots) 
can be hierarchized and can lead to a really effective storytelling program. The system also 
implements directional ultrasonic loudspeakers, capable of containing noise pollution in the 
museum by limiting the sound flow to the single users. Feedback in application testing re- 
ported a very comfortable perceptual experience. The novelty of this approach lies in the 
alchemical engineering of standard technologies, widely available, but not yet applied in cultural 
heritage environments, to achieve shareable, but very effective, results at low cost. In this case, 
it is of great interest the integration between motion recognition technologies with the pro- 
grammability of the software capable to create a single environment for authoring multimedia 
contents of high semantic value, dependent on the tactile interaction between the object and 
the blind perceiver, thus creating a real virtual/analog bridge independent of the technological 
mediator on the end-user side. 


Conclusions 


The project is currently concluded in its prototype state and has already produced many 
positive feedbacks, even in this early stage, developing a profitable process that has brought 
together public institutions and private initiatives in a very effective synergy, characterized 
by a remarkable transversality. 

One of the main results of this synergy was the positioning of the idea itself at the base 
of a further project included in the so-called 'Living-Lab Program’, bottom-up initiatives 
financed by the European Community, directly arising from the territories as long as certain 
issues are perceived as strong, positive instances by the communities. A consortium has 
been therefore established between University and private companies for the industrializa- 
tion of the prototype, within a more general enhancement action of the Museum and the 
Archaeological Park, that is taking shape in these very days. 

The project, in this new phase, has led to the creation of an Association (ATS) between the 
PAU Department [3], which participates as a Research Body, and private companies active 
in the sector of protection, promotion, and safeguarding of cultural heritage. 
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In the future, we plan to integrate into the system motion sensors with capacitive proximity 
evices (NFC, etc.), also through the field testing of new sensors based on the very high trans- 
duction capacity derived from the use of Graphene materials in the surface treatment of analog 
3D printed models, aimed at maintaining the hands-free approach that was greatly appreciated 
in the testing phase in a non-Immersive environment. 

Beyond the specific contents of this projects, and of the new ones, we strive to combine inno- 
vation, scientific rigorous approach, and enhancement of cultural heritage through an ‘inclusive 
attitude’, where the word ‘inclusion’ is purified of any declination linked to the idea of the ‘due 
by law’, or, even worse, to the one of ‘charitable intervention’. Cultural inclusion, and generalized 
fruition capabilities, on the contrary, must be intended, as they are indeed, directly connected to 
the economic strengthening of initiatives and territories and a powerful drive for development. 

This approach presents itself immediately as highly sustainable, not just being self-financed, but, 


+ 


thanks to the strong idea of transparent (and inclusive) commonality of all the project revenues, it 
can directly function as a medium for the transfer of technological innovation (often invoked, but 
rarely actually implemented) and generates a condition that we like to call ‘social empowerment’. 
The entire procedure becomes immediately shareable and the methodology can also be at 
disposal of “limited size” cultural institutions, such as small museums, or even private collection, 
which are, on the other side, often bearers of the highest historical and cultural values, partic- 
ularly if, together with their hosting communities, they would reach to build some territoria 


networks of high local identity values. 


o 


Notes 


[I] Rotary International — Disctrict IT-2100: D.G. 2014-2015 prof. Giancarlo Spezie; D.G. 2017-2018 dott. Luciano Lucania. 
Rotary and Rotaract Club Reggio Calabria and Nicotera-Medma. Special thanks to: arch. Salvatore Patamia, MiBACT; arch. 
Pietro Vicentini (Terna); ing Giusesppe Fedele (UICI Reggio Calabria); Prof. Giuseppe Lacquaniti (Historian and Journalist). 


[2] The Project has bee carried out by a group of young resources selected by public evidence from the Rotaract area and the 
Mediterranean University: Angela Balestrieri, Gabriele Candela, Barbara Cusato, Giuseppe Cutrupi, Roberta De Clario, Fabio Panella, 
Danila Punturiero, Verdiana Quattrocchi and Peppe Sorrenti. The operational coordination has been performed by Technical Tutors 
with consolidated experience: Antonio Gambino, Marilü Laface, Andrea Manti, and Roberto Prampolini for the web content. 


[3] The ‘EcoMedma’ consortium consists of Ecolandia SCARL (Leading Company, Pres. Prof. Antonio Perna), The PAU 
Department of the Mediterranean University of Reggio Calabria (Research body, Dir. prof. Tommaso Manfredi), CADI SRL 
(Dir ing. Piero Milasi). 
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REPRESENTATION CHALLENGES 
Augmented Reality and Artificial Intelligence in 
Cultural Heritage and Innovative Design Domain 


The Emotion Detection Tools in the 
Museum Education EmoDeM Project 


Paola Puma 
Giuseppe Nicastro 


Abstract 


Facial recognition technologies, already used nowadays in many applications, i.e. to support security 
systems in sensitive buildings, could in a short time achieve widespread use also in others sectors 
including culture institutions like museums or art galleries. 

The state of the art in the field of facial recognition allows discriminating factors not only related to 
the essential somatic characteristics of a person to recognize, with an ever-greater degree of precision, 
the emotional reactions that may occur on person's face. 

The article intends to describe the research's EmoDeM experimentation in the museum environment 
in order to provide a tool capable of interpreting the reaction of a user in front of an artwork and 
propose a responsive information content coherent what is manifested through facial expressions. 


Keywords 
digital heritage, museum education, machine learning, image classification, emotion detection. 
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The most important national and international cultural strategies clarify the importance of 
actions that include the use of digital languages and virtual environments allowing visitors 
to better understand the informative features about the observed objects and contexts. 
The hybridization of traditional spreading methodologies with the tools offered by contem- 
porary digital languages in the field of cultural heritage education therefore becomes an 
essential asset for those who work in the field of spreading of tangible and intangible values 
of the cultural heritage [COE 2005; New European Agenda for Culture, European Commis- 
sion 2018]. The most innovative aspect is not solely in the technological advancement of the 
supports used to collect, record and transmit information: the kind of digital contents, their 
being an informative material whose language has a communicative shape with potential 
and effectiveness only partially investigated, makes their experimentation absolutely cen- 
tral in the contexts connected to education and enhancement like museums and cultural 
heritage institutions. In fact, digital data has taken on a role not new but certainly enhanced 
in the current meaning of cross-media contents and effective integration with traditional 
museum educational tools [COE 2018; Work Plan for Culture 2019-2022; Council 2018]. 
The experimental use of new media and A.l. in the EmoDeM project represents, from this 


point of view, an attempt to further advance in the dissemination of cultural heri 
integral pivot of society and human knowledge. 


+ 


lage as an 


Field of the Research 


The EMOtion DEtection Museum project — EmoDeM concerns a study using some Al ap- 

plications in the museum environment to provide valuable indicators for museum education. 

Among the multiple thematic axes of the project, also relevant in negative terms, some can 

be identified as priorities: 

e the cultural axis relating to the EmoDem's human-centered characterization of ‘Al for 

society’ as explicitly pursued by the EU “Between ‘Al for profit’ and ‘Al for control’, Eu- 

rope could embrace ‘Al for society’, a human—centred, ethical, and secure approach that 

is true to our core values” [Annoni et. al. 2018]; 

e the scientific axis of the multidisciplinary concept focused on the Survey discipline 

and its various articulations: Architectural Survey (as a tool for acquiring and rep- 

resenting data for the knowledge of historical contexts), visual communication and 
digital storytelling; 

e the axis of the regulatory policy concerning privacy issues, the governance of the data 
sharing ecosystem and the management of information. 


Aims of the Research 


The article illustrates the early stages of the EmoDeM study consisting of the application 
of facial recognition technologies to museum education [Li, Li 2020, pp. 1-25; Baraldi et al. 
2015, pp. 2705-2714]. These Al functions are used to detect and interpret the facial expres- 
sions of the visitor in front of the exhibited artwork to propose a content corresponding 
to what is manifested [Artstein et al. 2014]. The project foresees that, by comparing the 
images acquired by an ‘intelligent’ shooting system in the immediate vicinity of the visitor 
with a dataset of pre-selected and classified images, it is possible to identify and classify with 
a good reliability the visitors reaction and consequently propose an information content 
identified in the connected and predetermined repository as ‘associable’ to the expressed 
facial expressions. 


State of the Art 


The state of the art in the field of facial recognition allows discriminating factors not only re- 
lated to the somatic characteristics of an individual: there are several experiments in which a 
software is instructed to identify the emotional states of the framed subject arriving at rec- 
ognizing, with a constantly increasing degree of precision, the emotive reactions that can be 
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manifested on the face of an individual (amazement, curiosity, attention, etc.) [Sharma 2019, 
pp. 834-837]. Facial recognition solutions that use Machine Learning programming are built 
on a sw models based on the analysis of large information datasets, called Training Datasets, 
and on the identification of recurring patterns inside it [|] the machine can therefore em- 
ploy what it has learned from the analysis of the datasets to making autonomous predictions 
and choices without explicit programming instructions [2] [Alpaydin 2020]. Research and 
experimentation in the field of machine learning have been demonstrating that the simula- 


aes 


tion performances of the h 
not always correct and relia 
nents of a specific problem, 


uman processes are currently very unstable as the detection is 
ble and yet the phases referring the identification of the compo- 
the comparison, the definition of a result, the refinement of the 


degree of reliability are in very rapid evolution [Alpaydin 2020]. This suggests that software 


of facial recognition current 


y used to support the security systems could soon also be used 


in different scenarios like museum education [Duguleaná et al. 2020, pp. 1-17]. 


Methodology 


The methodology used in EmoDeM integrates a workflow of operations divided into three 
macro-phases: |) acquisition of facial expressions; 2) image analysis and emotion detection; 3) 
response of the software through the matching of coherent contents. The first phase of the 
process has been the data acquisition using a video camera placed near an artwork, used to ac- 
quire the visitor's facial expressions [Fyffe et al. 2016; Legendre et al. 2018, pp. |-2]. The captured 
image has been subsequently processed therefore starting the emotion detection sequences. 
This phase certainly represents the most relevant and innovative part of the application as uses 
specific Machine Learning solutions responsible to evaluate the facial expressions detected and 
the visible visitor's reactions. The purpose of a machine learning program is to provide a comput- 
er with an algorithm consisting of a series of elements that can be compared in order to establish 
general criteria to learn from and to rely on in subsequent queries. For this purpose, it's essential 
to develop datasets consisting of many specific data including the significant characteristics 
be correctly interpreted from the software during the preliminary training phases. Therefore 
enhance the sw detection, in addition to the images directly acquired from the authors duri 
the test, EmoDeM will employ specific datasets expressly designed for emotion detection appli- 
cations, i.e. CK+ [3][Lee, Kang 2020, pp. 15-27]. In its complete configuration the last output of 
EmoDeM is the proposal of relevant contents coherent to facial expression. 


First Results 


The technical solutions used in the EmoDeM study have been inspired from the results obtained 
in a previous research on the documentation and spreading project about the abbey of Badia a 
Passignano [Nicastro 2020]. At the end of that experience, an enhancement project was drawn 
up whose Al images detection functions have been later integrated into the EmoDeM study: in 
the first research an application was developed to provide information for the visitors of Badia 
a Passignano [4]. For intrinsic reasons, the architectural components are, however; characterized 
by visual patterns that are much more easily typified than those characterizing a totally inhomo- 
geneous and variable set such as peoples facial expressions; this constitutes the major limitation 
of the test, illustrated below also in the failure aspects. At present, the experimentation phase 
relating to detection in a simulated environment was held, the purpose of which was to verify 
the degree of confidence of the software in the correct classification of the apparent reactions 
shown by the faces of people photographed in front of a pictorial work. For this purpose, a set 
was set up in which 6 participants were invited to observe three different reproductions of pic- 
t kstation was used to capture the images of facial expressions, catalog 


torial works. A smart wor 
and analyze them with the Google Cloud Vision API software. Once the acquisition phase was 
completed, the subjects involved were asked to fill in a questionnaire in order to explicitly indi- 
cate the reactions felt during the test, then comparing the answers with the answers provided 
by the software. The analysis of the acquired data reported that the matching on 18 answers (6 
participants examined on three works) between classification operated by sw and by question- 
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naires resulted in: 9/18 occurrences of ‘too low match’, 5/18 occurrences of correct outcome, 
4/18 occurrences of incorrect recognition. The phases relating to the training of the platform are 
therefore already partially underway, and the first application test of EmoDem is expected to 
be carried out in a short time in a museum environment. The on-site installation will allow the 
feasibility check to evaluate the efficiency of the concept, and to expand the dataset of images 
collected in different contexts than laboratory experimentation, and test the activation of the 
response by proposing coherent contents. 


Conclusions 


Although the research is in its initial stages, the tests carried out in the laboratory have 
highlighted the first technical critical issues, especially about the images to be used in the 
sampling phases of the faces. In fact, the most significant advantage of pre-compiled dataset 
consists of using images suitably prepared to be used during the training phases of the soft- 
ware; on the other side, the images acquired in a controlled research environment present 
a certain degree of theatricality in the emotions expressed and a lesser naturalness. 

In the museum workfield phase, in order to improve the responses of the application, a 
second training dataset will have to be compiled with an expanded number of images with 
greater expressive naturalness. The major general problems we have encountered so far 
using Al tools are of two types. The first one is clearly represented by the skills, especially 
in computer programming languages, necessary to profitably conduct research and exper 
imentation using Al applications; however, it should be noted that the issues relating to Al 
and machine learning systems are nowadays at the centre of public debate, and explains 
why in recent years the available tools and documentation have multiplied and its possible 
operate with these tools with an appropriate level of awareness in a multidisciplinary team. 
The second threat is represented by the regulatory policy and visitors privacys protec- 
tion issues, which must be defined with law criteria to allow the activation of information 
support only for consenting visitors. In the feasibility analysis of the project, the use of AR 
solutions could appear as a more effective choice than the efforts required to use an im- 
age recognition algorithm. The advantage in this last case is related to the fact that unlike 
the AR targets solutions, in which the references to the information contents are defined 
in the design phase and remain so once defined, in this mode the information is provided 
contextually to the reactions of the observer without there being a preordained sequence. 
EmoDeM tries to shift attention from the ever—increasing production of information arti- 
facts to the adoption of technological solutions that can assist museum curators in propos- 
ing as much as possible inclusive and customized informations. The technological advance- 
ment in the field of Al therefore gives us the possibility of operating in the space between 
the physical object and the related digital contents, with respect to which the concept of 
EmoDem has been set up, designed precisely with the function of interface between the 


user and the digital information contents. 


The editorial and scientific responsibility of the chapters is recognized to: Paola Puma for 
ntroduction, Design of the research, Aims of the research, Conclusions; Giuseppe Nicastro 
for State of the art, Methodology, Expected results; both the authors Paola Puma and Gi- 
useppe Nicastro for the figure and References. 


Notes 


[|] These images must be suitably prepared to be used in a Machine Learning workflow: in a first phase, the content of each single 
image will be divided into cells, according to a SxS grid, containing a defined number of pixels. Subsequently, the contents of any 
cell will be examined to identify classes of pixels with similar characteristics in areas defined by appropriate tags. The datasets thus 
prepared will therefore be used in the software training phases, or when further images, not belonging to the datasets used, will be 
analyzed to identify and classify their content. 


[2] Using Machine learning solutions and computer networks in the analysis of large datasets represent practices becoming 
increasingly widespread. Companies like Google or Amazon offer today increasingly affordable solutions for those who intend 
o approach these technologies: if until a few years ago, developing an experiment in the field of machine learning implied to 
access to computer networks available only to specialists and researchers, today Cloud Computing has simplified the access 
o these resources. 
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[3] Among these, one of the most used is CK + [Lee, Kang 2020, pp. 15-27], an archive of images acquired in a laboratory 
environment and catalogued in frame sequences with which it's possible to investigate different types of faces during the 
gradual transition from a neutral emotional state to one of the seven emotions available (anger, disgust, sadness, happiness, 
amazement, contempt and fear). 


[4] The Passignano abbey, located in Italy between Florence and Siena, has been documented by digital survey that used the 
integration of data acquisition methodologies to document the consistencies and architectural features of the complex for the 
purposes of knowledge and cultural enhancement. In this case, the operational scenario included the LS survey, the topographic 
survey, and terrestrial and aerial SFM. At the end of the research, an enhancement project was drawn up and application was 
developed to provide information support for visitors. BAPP application uses the Badia a Passignano complex and its works of art 
igures acquired during the survey campaign to train the image recognition software. By this procedure Bapp makes it possible to 
frame with the camera of your device any portion of the complex located along a defined visit path to access contents consistent 
with the object framed by the camera attaining the output deriving from the machine learning processing. 
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Abstract 


The MareXperience exhibition event was held in Procida on February 22 and 23, 2019, inside an 
ancient cave for the shelter of fishermen's boats, 45 meters long and 8 meters wide. Inside, synchro- 
nized video projections and soundscapes were produced to create an immersive virtual environ- 
ment, whose contents were inspired by the signs and fluidity of underwater life. The digital forms 
were generated by a Swarm Intelligence algorithm, a tool generally used to create complex kinematic 
structures defined by a large number of parameters. Generative design is an expressive form aimed 
at artistic languages and representation of dynamic contexts, which here have been integrated into a 
narrative process, connecting aspects of the recording of the movement of bodies in water with the 
principles of data visualization. 


Keywords 
artificial intelligence, underwater, video projections, generative design, data visualization. 
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Theoretical Principles 


The digital has extended the ordinary scenarios of research on the languages of the repre- 
sentation of reality, opening it to the complex nature of events, including it in the dynamic 
mechanisms of the permutation of forms to describe the variable nature of contexts; it has 
extended the boundaries of drawing, crossing the limits of the languages of representation 
and finding itself close to the matrices of thought, to its operating mechanisms before words, 
the text, stop their sense. The extensive fields of artificial intelligence, which now support 
innumerable actions and choices we make on a routine basis, are becoming more and more 
widespread, but solutions and tools often show the total absence of the critical processes 
that should support the evaluation phases, marginalised by improper technicalities in the 
service of the logic of profit. Yet the virtual was based in thought before taking shape as 
the result of technology, and in thought it defined the extensive space of the configurations 
of the real, of the possible, which underlie the variable nature of augmented reality. From 
philosophy, from Leibniz, from Deleuze and others, those coexisting paths were traced 
which lead to Turing and Gédel, to the principles and machines that have made possible the 
current state of technological research. 
The concept of the virtual, as defined by Gilles Deleuze, extends the real by opening it up 
to dynamics of variation, the actualization of which imposes an increase in dimensions and 
a consistent alteration of temporal sequences [Deleuze 1997; Levy 1997]. This structure 
requires a complex place to give shape to the images of thought, a virtual space must be 
traced that includes matrices of relationships between values, meanings, expectations and 
emotions, as well as rooting it in the perceptive structures of reality, to facilitate the intro- 
jections of messages [Repola 2008]. 


The Project 


The project tested a methodology for recording bodies in motion, transferring kinemat- 
ic parameters into a digital simulation environment, modulating trajectories according to 
tension fields consistent with the compositional patterns of a virtual space overlapping 
the cave [Repola 2018, pp. 781-788]. The project started from research activities carried 
out at the Department of Humanities of the University Suor Orsola Benincasa and the 
Sebastiano Tusa Civic Museum of Procida, aimed at developing systems and procedures for 
the three-dimensional survey of seabed and submerged cultural heritage. Real numerical 
models offer new opportunities of spatial data management for the analysis of places and 
the simulation of possible scenarios and complex events. Ihree-dimensional data, moreover, 
are well adapted to be used in parametric modeling procedures for the development of 
immersive museums, where scenarios can change according to the needs related to the 
data representation. 
The MareXperience project aimed at verifying a series of interrelation schemes between 
different digitization procedures of real contexts and solutions for data visualization in an aug- 
mented space, given by the overlapping of a segment of underwater life and an ancient cave 
for the shelter of fishermen's boats on the Silurenza beach on the island of Procida (fig. |). 
In the weeks preceding the event, several underwater video recording sessions were carried 
out in the Pizzaco and Solchiaro areas to record the movements of schools of fish. Subse- 
quently, the acquired videos were processed in order to provide the numerical parameters 
necessary to generate the digital animation that composed the artistic performance. In addi- 
tion, underwater soundscapes were recorded at different depths through the use of a hydro- 
phone. The sound of the sea and its life became the track on which the three-dimensional 
digital sound integrated with the animations was processed. The latter allowed to place, in a 
virtual way, sound sources in the space increasing the perceptive levels of the visitors. 

The immersive environment [Dede 2009], inside the cave, was realised thanks to a tech- 
nological system built to project on the long vertical surfaces of the cave the digital video 
and audio processing produced in the previous phases. In particular, this system consists of 
a media server, specially assembled for the event, which separated and distributed, in syn- 
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Fig. |. Point cloud of the 
cave. 


chronized mode, both the video stream, to 4 ultra-short-throw video projectors, and the 
spatialized audio stream with 5 independent channels, to the respective 4 active acoustic 
speakers and the subwoofer 

The immersive installation project has exploited the spatial compression of the place, due 
to the oblong shape of the cave, to generate in the visitors the unexpected perceptive 
experience of extension of the places through the movements of the bodies beyond the 
limits of the long walls. The underwater soundscape, thanks to the techniques of frequency 
modulation and variable distribution of sound intensity on a spatial basis, supported the 
depth and three-dimensionality of the simulated space generating an adequate level of 
immersiveness of the installation. 


Software Procedures 


The research project, aimed at the construction of a representative protocol able to relate 
the user and the cultural product through a non-linear narrative logic, used technologies as 
a tool for the representation of aesthetic language, using machines not as a final output of 
representation but as a communicative vehicle. The overlapping of the levels of interaction 
between tools, software and communication languages, from the earliest stages of compo- 
sition, anticipated the ordinary relationship between machine and man, which places one 
subordinate to the other The primary cause of iteration was the generative principle of 
movement in nature, which recorded in a marine life interval, broken down and sequenced, 
has become the code of development of the kinematics of animation. 
The possibility of having a large amount of data to analyze, has allowed the structuring of a trans- 
versal strategy of elaboration of the audiovisual content. For this project we used the principles 
of computational analysis for clustering data in a three-dimensional environment, taking as input 
parameters only the position attributes derived from the data extraction process. In this phase 
we provided the algorithms with a series of data samples chosen from the cluster categories, ob- 
served the results by manually correcting the parameters with respect to the expected aesthetic 
configuration and as consistent as possible with the processes of relation between real and virtual. 
The Swarm Intelligence algorithm [Bonabeau, Dorigo, Theraulaz 1999] was used to generate 
the kinematic flows. Based on the analysis of the mass movement of fish, it is able to simulate 
the optimization principles of the movements of marine animals within the social behaviors 
during migration periods, and in strategies aimed at finding food and self-protection. 
This algorithm is therefore one of the best approaches for the realization of complex kine- 
matic structures composed of numerous agents, the considerable advantages such as high 
speed convergence, flexibility, fault tolerance and high accuracy, are essential and unavoid- 
able values for the synthesis of a container capable of metabolizing the enormous amount 
of data collected during the previous phases. 

Touchdesigner [|] was used as the synthesis software for this first test phase of the data in- 
tegration method to manage the emission and control processes of particle elements in real 
time. The software manages these entities by means of ‘SOPs’ (surface operator families) which 
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Fig. 2. Software 
procedures. 


Fig. 3. Video editing 
phases. 
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guarantee the parametric connection between numerical entities with motion streams. The Sur 


face Operators family, or SOPs, is useful for all 3 


D operations, whether they refer to simple 


three-dimensional geometries, particle systems, architectural models, or 3D characters. For the 


optimization of rendering and thus visualization o 


f real-time graphics processing, it should be 


remembered that SOP transformations occur on the CPU, which must be performed for each 
vertex in the geometry, taking up a lot of resources; instead, component—level transformations 
are applied directly to the 3D geometry, or object, as a whole and are computed on the GPU 
as a single operation.A single operation performed on the GPU is definitely preferable to what 
could be hundreds of thousands of operations performed on the CPU. The total number of 
points, primitives, vertices, and meshes will vary depending on which model is being processed, 


but the basic principle is that the more polygons/vertices there are, 


+ 


the more computing power 


and graphics memory will be required to complete the tasks. The project verified the possibility 


of connecting such SOPs with the Swarm Intel 


igence algorithm to include in the 


kinematic 


generation processes aspects of the natural movement of marine living beings. Connecting the 
calculation matrices to the SOP operator, the data of interest are segmented to extract numer 
ical parameters inherent to the management of their position, rotation and scale on the three 
xyz axes. These data are linked to the Instance of the geometry containing the 3d source model 
of the particle system. The geometry instances in the Geometry COMP are copies of the object, 
which can be transformed independently. In fact, it is possible an instance for each sample of a 


+ 


CHOR row of a table, pixel of an image or point 


of a SOP In this way, each individual particle 


is a 3d model with its own levels of automatons. In this experimental design phase, the entire 
particle group is modified in the physical simulation parameters, specifically turbulence and wind, 
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by realtime audio analysis. (fig. 2) The aesthetic result obtained from the verticalization of the 
swarm intelligence algorithms and the vector fields obtained from the data extraction process 
is based on a continuous reiteration of the same data, which are influenced again at each frame, 
generating a swirling kinetic movement; the nodes of which the algorithm is composed generate 
intelligent feedback signals that mimic the social behavior of marine animals, elevating a behavior- 
al characteristic to an aesthetic parameter: 


Results 


The ‘sea experience’ is the paradigmatic object used to build a model of representation of 
a dynamic reality, such as the underwater one and its fauna, through the application of fluid 
architectural constructs rendered through the methodologies of generative digital design. The 
connection of the two realities, the original and the disguised one, has obtained by means of 
a rigorous process of acquisition of the real data, which underwent a process of data—analysis 
and data-extraction to be then processed with artificial intelligence algorithms that defined 
the formal and expressive rules of the virtual artefact. 
The analytical model of information extraction took into account the different dimensional 
metrics typical of a living environment, such as the underwater one, synthesized by means of 
the three-dimensional vector trajectories made by a school of fish, their speed of movement 
and the acceleration index in space. The goal was to re-generate a non-mimetic reality of 
the original by designing a representative complex, with different levels of reading, to stimulate 
the ‘emotional understanding’ of the context investigated, in an attempt to bring the level of 
knowledge to the stage of wisdom (Ackoff's model). The Exhibit design, rendered through 
video mapping integrated with underwater soundscapes, produced an integration between 
reality and a narrative component such as to stimulate the imagination, in the sense intended 
by Bachelard with the term réverie [Bachelard 1973], and the synchronic perception of the 
different environmental qualities referable to the places of the sea and the cave. 


Notes 


[I] The latter is a Python programming environment in which you can visually manage user actors or operators with specific 
tasks that are linked together to create audiovisual patches. 
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Abstract 

The paper presents the first results of an ongoing research on the benefits of implementing com- 
putational modeling in Scan—to—BIM processes for the representation of historic architecture in AR 
and VR application with education and communication purposes. The 3D laser scanner survey of a 
complex vaulted system was the starting point for the development of the research. The design of an 
App that optimizes the effectiveness of computational modeling concludes the workflow exporting 
the FBX file from Revit and then importing it directly onto the Unity real-time development platform. 
The study shows the feasibility of the computational approach in the application of AR and VR sys- 
tems demonstrating its effectiveness thanks to the segmentation and hierarchization of the different 
components of the ribbed cross vault. 


Keywords 
interactive AR, scan-to-BIM, algorithmic modeling, 3D visualization, mobile applications. 
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Fig. |. Workflow from the 
survey to the mobile AR 
app (left); Point Cloud 
from the Laser Scanner 
surveys (right). 


3D Digital Representation as Effective Communicating Framework for Cultural Heritage 


Information and communication technologies (ICT) have expanded the dimensions of survey 
and representation models in architecture. Representation systems and graphic schematiza- 
tion have always been a crucial step in architecture learning process. Only due to a correct 
coding of the visual message, the abstract conventional 2D models of architectural drawing 
convey information in an immediate, rich and effective way. A successful graphic represen- 
tation needs to assume a certain distance between the reality and its visual exemplification: 
communication acquires immediacy and quality as the degree of schematization and coding 
increases. In some cases, especially when there is no sharing of the codes of representation, 
only limited information and knowledge are available about traditional 2D graphic models. It's 
now widely proven as ICT can favorably impact student learning and communication of Cul- 
tural Heritage (CH), as for example in teaching-learning interaction, museum installations or 
virtual and augmented visits [Mortara, Catalano 2018]. 

New technologies such as 3D laser scanning (LIDAR) or Structure from Motion (SfM) sys- 
tems constitutes a valid support to cultural heritage visual presentation and documentation 
in different fields of scientific research and professions. Despite the interpretation and critical 
representation of the massive amount of 3D data of these two systems sometimes becomes 
a limit, these technologies can certainly express strong potential also for communication and 
educational purposes. A consolidated application in architecture is certainly that of the so— 
called Scan-to-BIM processes aimed at generating semantically rich as—built BIMs of com- 
plex objects from 3D point clouds datasets. In our research, computational modeling allows 
an approach that registers BIM components with semantic information driving the design 
process from both 3D point clouds data and rules from historical architectural treatises. By 
parameterizing the modeling process, the algorithms prove to be particularly effective in the 
analysis and representation of object categories normally non-native in BIM libraries. The 
algorithm also presents the advantage of recording information on the segmentation and 
hierarchization of the different architectural components, with interesting and useful reper- 
cussions in AR applications. In addition to the methodological contributions, computational 
approach allows us to interact in the visualization of the different individual components and 
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to have a range of possible relations already stored in the algorithm. The workflow was pro- 
totyped in parametric design with Visual Programming in Dynamo with Autodesk Revit and 
tested on a 3D point cloud of the XVI century church of Nostra Signora della Speranza in 
Cagliari (Italy), scanned via Leica Laser Scanner HDS 7000 (fig. |). Starting from the 3D laser 
scanner survey of the entire monument, the workflow was tested by focusing attention on 
the main components of its complex vaulted systems, consisting of some ribbed cross vaults 
with pointed arches. To create smart data queries and visualizations, an App design process 
optimizes the potential of the approach by completing the workflow, enhancing the different 
visual analyzes in a very intuitive manner. AR and VR technologies provide new ways of space 
visualization and architecture interpretation, promoting and developing building reading skills. 
Thanks to the design of specific tasks aimed at users such as first-year architecture students, 
the App aims to optimize the reading of the anatomy of the ribbed cross vault and the 3D 
spatial relationships of its different architectural elements. 
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Fig. 2. Classification in 
Uniclass 2015. 


Algorithmic Modelling in AR Applications 


For years now, the procedural generation of 3D models has played an important role in 
architectural production. The possibility of creating parametric processes capable of remod- 
elling a three-dimensional element in a largely automated way allows the definition of even 
very complex rules for the generation of elements [Tedeschi 2014]. 

This type of approach is particularly interesting when dealing with elements with particu- 
larly complex geometries and at the same time constrained by the need to correspond as 
closely as possible to the physical models to be represented. An emblematic situation is that 
of Scan-to-BIM processes applied to complex historical elements, like gothic vault systems; 
these elements are in fact very often absent from the normal native libraries of BIM tools, 
and their complexity highlights the numerous constraints that BIM environments present in 
terms of modelling, since they are by nature information modellers and not 3D modellers 
[Argiolas et al. 2019]. 

The use of development environments based on Visual Programming Languages (VPL) has 
made it possible on several occasions to get around the deficits of the modelling tools of BIM 
environments, by releasing the modelling process from the single element, and linking it to 
the category of elements and therefore to their invariances [Bagnolo et al. 2019] (fig. 1). In 
the case of gothic vaulting systems, the hypothesis has been put forward that the definition 
of the geometry and curvature of the ribs is sufficient to define the entire vault [Willis 1910], 
to the extent that in certain geographical contexts materials and techniques do not lead to 
noteworthy stylistic variations [Agustin-Hernändez et al. 2018]. 
This is possible thanks to the use of modelling algorithms developed by means of special 
environments such as GrassHopper for Rhinoceros or Dynamo for Autodesk Revit, both of 
which are now highly integrated with Revit; this makes it possible to keep the entire model 
creation phase within the BIM environment, significantly limiting the problems that can arise 
from the passage of data between software through export/import of models. 

The algorithms also represent a container of information regarding the model, constituting a 
real metadata of it; in the algorithm, in fact, information on the geometric rules used for the 
realisation of the model components remains accessible, as well as how these components 
are assembled together and therefore, what is the hierarchical organisation of the model. 
The use of augmented reality technologies is now an established practice for the communi- 
cation of architectures, whether they are just designed or built, especially for the historical 
heritage [Spallone, Palma 2020]. It is possible to see how there is a bidirectional link between 
BIM models and augmented reality for the communication of architecture; if it is true that 
the immersiveness offered by AR plays a fundamental role in the process of ‘telling the story’ 
of architecture, it is equally true that the organisation of the elements and their classification, 
typical of the BIM methodology, offers a further level of deepening and understanding of the 
building organism. 

In our specific case, notwithstanding the considerable impact of the mixed visualization of 
the model as a whole, the possibility of breaking down the various elements in real time and 
managing them autonomously in their graphic representation, offers an enormous expansion 
of perspectives of use. Even the mere hierarchization of the elements, in the case study 
based on the idea of the Uniclass 2015 classification [NBS Enterprises Ltd 2021], allows us 
to offer the user a reading by levels of the objects, facilitating their comprehension (fig. 2). 
Another advantage can be derived from the implementation of the logic of levels of detail 
(LOD) of the BIM models, which lends itself well to adapting the amount of information dis- 
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Fig. 3. View of the AR 
mode with the hide/ 
unhide feature. 


Fig. 4. View of the preset 
drawings mode (left), and 
the detail screen (right). 


played and its degree of precision. This translates into a differentiated visualisation according 
to the level of “zoom” with which the user observes the objects. 

The implementation of the model of the Church of Our Lady of Hope within AR scenes was 
carried out using the real-time development platform Unity [Unity Technologies 2021], for 
the design and programming of all the interfaces and mechanics of the app. In addition to the 
standard modules, Vuforia [PTC 2021], a free package for the creation and management of AR 
cameras and targets, and PCX [Takahashi 202 1] for the import of point clouds were added. 


Exploration and User Interface Design 


The software as configured, allows the possibility to live two main types of experiences, one 
in augmented reality in third person and one in virtual reality in first person. 

Once the application is started and the ‘start’ button is pressed, the software enables the 
camera and the device is ready to recognize a target which can be: 

* figurative, as in our case study in which the building plan was chosen; 

* coded, such as OR Codes. 

Once the target has been identified, the 3D mesh model appears on the screen and, at the 
same time, the control panel located in the lower part of the interface is enabled, allowing 
you to activate various functions. Hide/unhide elements (fig 3). Pressing the ‘layer’ button 
opens the sheet showing the elements according to classification system based on Uniclass 
2015 of the building organization. The software allows you to turn them on and off individ- 
ually with the relative box. Pressing on the text item switches instead to the detail screen, 
which specifically analyses the element (fig. 4). Organization in pre-packaged documents. 
Clicking on the square allows you to view the model according to the predetermined draw- 
ings such as plans, sections and axonometric views (fig. 4). General information. The ‘i’ button 
takes you to a tabbed screen on the historical information of the building which also recalls 
a photographic section. The browse button allows you to switch from augmented reality 
to virtual reality mode; starting from a first-person view, typical of some video games, it 
is possible to virtually move in the external and internal space of the building thanks to the 
aid of two virtual pads, one dedicated to the movement of the camera, the other to the 
movement in the model. In this navigation mode, thanks to on the PCX package, it is possible 
to switch from navigation on the polygon mesh model to the point cloud produced by the 
survey with the LIDAR, which allows a comparison between the two products and an even 
more immersive detail view. 
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Conclusions 


Representing the meaning embodied in complex works of architecture like vaulted systems, 
this approach allows a novel way to interact with the history and the theory of architecture. 
The combination of AR technology with the use of touch screen technology of mobile devic- 
es makes the manipulation and visualization of semantically-enriched 3D models very simple 
and intuitive. Interaction can take place either through QR codes or printed images like 
architectural plan. Improving 3D scene synthesis systems, the app allows a range of semantic 
queries into 3D model datasets. Strictly depending on the characteristics of the specific cate- 
gory of architectural components, the features included in the app prove to be very effective 
in the communication and representation of architecture. The research plans to continue 
by implementing in the 3D model all the constructive elements of the different parts of the 
church as well as sacred art and sacred furnishings of historical and cultural interest. 
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REPRESENTATION CHALLENGES 
Augmented Reality and Artificial Intelligence in 
Cultural Heritage and Innovative Design Domain 


Integrated Technologies for Smart 
Buildings and PREdictive Maintenance 


Marcello Balzani 
Fabiana Raco 
Manlio Montuori 


Abstract 


The preservation and the regeneration of the existing built heritage is still characterized, even in 
the context of an increasing digitization of the value and supply chain, by inefficient time and costs 
management, along the whole life cycle, as well as by discontinuity and lack of information on the one 
hand, by redundancy and duplication of data on the other Accessibility, usability and feasibility in order 
to univocally implement information, also by real-time monitoring, are areas of growing interest to 
all actors of the building and construction value-chain, with particular reference built heritage knowl- 
edge phase, as well as for the stakeholders of complementary industries as ICT, for the development 
of integrated digital solutions for data acquisition, modeling and visualization. 


Keywords 
information modeling, digital documentation, information visualisation, built heritage, KETs. 
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Fig. |. Project 
requirements: multiplicity 
of data sources 
characterized the 
intervention on the built 
heritage. 


Introduction and Overall Framework 


The InSPiRE project, Integrated technologies for Smart buildings and PREdictive maintenance 
[I], is funded under the Smart Specialization Strategy of Emilia-Romagna Region, call for 
“Strategic Industrial Research Projects” 2018 [Hegyi 2021], and implements the architecture 
of a predictive diagnostics system for monitoring the state of preservation of materials, com- 
ponents and systems of the existing built heritage that, under normal operating conditions, is 
approaching the end of its useful life. At the end of the first year of the activity the project is 
facing several challenging actions aimed at wide spread data integration and sharing, starting 
from a methodological and technical advancement in built heritage digital documentation. 

As part of digitization process applied to built heritage as well as to Cultural Heritage, digital 
documentation and visualisation of the whole building life cycle, from project phase to facil- 
ity management phase, is emerging as effective strategy in order to support both decision 
making and sharing of information from different data sources: shape and morphology; diag- 
nostics; safety in use; risk management; maintenance. Implementing collaborative real-time 
monitoring platforms based on enabling technologies, such as sensor networks, is one of the 
main goals of the InSPiRE project. 

Morever similarly to the intervention on the cultural heritage, the intervention on the built 
heritage is characterized by: multiplicity and variety of information sources, with reference 
to different periods of the building life cycle; lack of homogeneity, and often absence, of or 
ganization and hierarchy of information; plurality of design purposes; multiplicity of methods 
of investigation and technologies applied; plurality of professionals involved [Garzino 2011]. 
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Digital documentation and visualization of the built heritage 


The overall project workflow is developed starting from requirements: what type of data 
characterized the project on the built heritage (fig. |); what kind of information the variety of 
professionals involved are asked to provide in order to manage effective decision making pro- 
cesses; with reference to the new ICT technologies applied to the project, such as BIM tools 
and methods, what kind of both opportunities and boundaries have to be taken into account. 
Through networks of wireless sensors, based on smartbrick technology, case studies of 
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public housing are placed in continuous monitoring in order to: implement monitoring 
systems based on Integrated Enabling Technologies (KETs); develop predictive algorithms; 
implement a platform for displaying and managing information, even real time data, sup- 
porting awarness during the preservation and the regeneration phases of the existing 
built heritage. 
The sensor network is currently being tested in the context of an ACER case study in 
Bologna, consisting of a 1950s block-type residential building. 

The acquisition of information from the sensor network and the management of the 
dataset originated from the diagnostic campaign by multispectral images are currently be- 


+ + 


ng implemented within the framework ofthe above-mentioned case, in order to identify 
trajectories of behaviour of the building system, in particular in the fields of structural and 
seismic safety. Starting from previous research results financed within the “Por Fesr Impre- 
sa 2015” call, the result of the InSPiRE project is a digital platform, currently under develop- 
ment, of strategic decision support for predictive maintenance and management activities 
that, by implementing intervention procedures on an existing built heritage in borderline 
and/or emergency conditions, increases its useful life and capitalises its economic value. 
The involvement of local companies — such as international leaders in the market for the 
production of materials and systems for the intervention on the existing buildings as well 
as in the management and processing of big data and in the development of advanced 
sensors —, which is a strategic action in order to support effective project results, regional 
and trans-regional cluster industrialization specialisation and value—chain innovation and 
competitiveness [European Commission 2019], favors the implementation of the specifi- 
cations of the architecture of the monitoring platform, contributing to the validation and 
demonstration in the relevant environment (TRL5-6), and to demonstrate the proto- 
types in the operational environment (TRL7). 
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Data Integration and Information Modeling 


The need for the implementation of strategic decision making tools has originated from 
the fact that there are both a variety of sources of information and methodologies regard- 
ing the built heritage documentation. 

As a result, the availability of so many active sources providing valuable information to be 
used by a variety of experts and actors of the value-chain determines whether data re- 
dundancy or loss of information when hierarchy, segmentation, accessibility, usability of data 
as well as built heritage nomenclature are not properly considered. 

The starting point in order to implement the InSPiRE digital platform is the data sources 
hierarchy with reference to the case studies investigated. Each partner deepens, in accor 
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dance with the project objectives, specific skills and pilot tests, a number of different types 
of source data such as: historical data sets related to the interventions of ordinary and 
extraordinary maintenance; satellite data for the analysis of the territorial scale; data from 
continuous monitoring for the analysis of the structural behavior of the building; data from 
integrated diagnostic campaigns for the evaluation of the energy behavior of the building; 
data from direct diagnostics for the analysis of mechanical and physical-chemical charac- 
teristics of materials in place. 

Subsequently, the data modeling phase and definiton of information hierarchy allow to 
define; trajectories of behavior for those phenomena that allow an approach to the eval- 
uation also of predictive type; criteria for the subsequent visualization, accessibility and 
usability of information, with reference to different categories of experts identified as end 


users of the project results [European Union 2018]. 


Toward Applications and Digital Protocols for Built Heritage Management 


So far, INSPIRE project aims at defining a data integration protocol for built heritage man- 
agement. From one point of view, the system responds to the request for timely anticipa- 
tion of the degradation and damage phenomena, by leading the maintenance actions in an 
adaptive way, with respect to the phenomenology of the degradation cause. Subsequently, 
the project makes use of the technological skills of industrial ecosystem of the regional ter- 
ritory, whose development can potential benefit from the collaboration during the entire 
ife cycle of the project. Definitely, the development of the different components of the 


monitoring architecture requires that platform integrates and cooperates with experti 


tise 
in the field of: restoration of both cultural heritage and built heritage; construction science; 
digital technologies applied to the survey and diagnostics phases; computer science; chem- 
ical and mechanical characterization of the building materials; integrated technologies for 
building preservation, retrofit and maintenance. The intersectorality and interdisciplinary 
nature of the project involve strategic sectors and disciplines that respond to the drivers 
of innovation identified by European Smart Specialization Strategy for the construction 
industry. The development of technologies for predictive diagnostics applied to facility 
management is a cross-sectoral challenge to the regional Clust-ER ecosystem, such as 
Emilia-Romagna industrial and research network, and to the European strategic objectives, 
relying on enabling technologies (such as new smart materials or the pervasiveness of the 
nternet of Things technologies) framed, specifically for the Emilia-Romagna Region, in the 
Clust-ER “Build”, “Create”, and “Green Tech”. 

According to the InSPIRE workflow the data integration protocol definition process start- 
ed studying what and how surveying data to be included in InSPiRE digital platform, as well 
as analysing how information to be modeled in order to achieve both interoperability and 
accessibility. Definitely data visualization is closely related to methodology of archiving and 
retriving digital information in order to make definition of common topology effective. The 
platform thus implemented will undergo a testing process by different target end users: 


professionals; managers of complex real estate assets; maintainers; facility managers. 


Conclusions 


Developing integrated digital tools for the management and visualization of information 
related to the intervention on the built heritage responds to the dual need, expressed by 
European policies and beyond, to: supporting the adoption of data—driven tools to make 
decision-making processes more effective, less expensive and more sustainable; promote 
the industrialization of the supply chain. 

Consequently, the objectives that the project pursues are closely related to the diffusion 
of H-BIM protocols for the existing heritage [Hung-Ming 2015]. In this sense, the results 
of INSPIRE are implemented by both objectives and results of other projects Por Fesr fund- 
ed, such as the eBIM project: existing Building Information Modeling for the management 
of the intervention on existing heritage. The definition and implementation of semantic 
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ontologies in order to organize the knowledge around the complexity of the intervention 
on the existing heritage cannot be separated so much from the definition of a common 
lexicon as from the correlation to the purposes of the intervention, rather than to the 
categories and typologies of the built heritage [Pauwel 2013]. 


Notes 


[I] The project “InSpiRE — Integrated technologies for Smart buildings an PREdictive maintenence” involves five partners, 
including national universities and research centers, such as: Laboratorio TekneHub, Tecnopolo of the University of Ferrara 
(Lead partner); CIRI EC, Interdepartmental Center for Industrial Research Building and Construction, University of 
Bologna; CRICT, Interdepartmental Center for Research and for Services in the Construction and Territory Sector of the 
University of Modena; CNR Istec, Institute of Science and Technology of Ceramic Materials; Flaminia Center for Innovation. 
Moreover, seven companies are part of the partnership, from the regional territory and beyond, with reference to: production 
of materials and components for the chain of intervention on the built environment; ICT products; enabling technologies; 
diagnostic services for the built heritage. 


[2] The project “eBIM: existing Building Information Modeling for the management of the intervention on the built heritage”, 
which has received funding from Por Fesr 2014-2020, involves five partners, including national universities and research centers, 
such as: CIDEA, Interdepartmental Center for Energy and Environment, University of Parma (Lead partner); Laboratorio 
TekneHub, Tecnopolo of the University of Ferrara (Lead partner); CIRI EC, Interdepartmental Center for Industrial Research 
Building and Construction, University of Bologna; Centro Ceramico; Certimac. Moreover, ten companies are part of the 
partnership, from the regional territory and beyond, with reference to: architectural and engineering firm; production of 
materials and components for the chain of intervention on the built environment; ICT solutions. 
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REPRESENTATION CHALLENGES 
Augmented Reality and Artificial Intelligence in 
Cultural Heritage and Innovative Design Domain 


Extended Reality (XR) and Cloud—Based 
BIM Platform Development 


Fabrizio Banfi 


Abstract 


Extended reality (XR), Artificial Intelligence (Al) and Building Information Modeling (BIM) for the 
digitization of cultural heritage are proving remarkably successful in different fields of application. 
The convergence of innovative methods, latest-generation technologies, and software applications 
for the representation, storage, transmission of tangible and intangible values of architecture, turn 
out to be increasingly decisive both in supporting the project's needs of the professionals involved 
in the valorisation and management of the built cultural heritage and in enhancing the transmission 
of computer-generated perceptual information for all types of users (expert and non-expert). For 
those reasons, this article presents research focused on the development of an open-source cloud- 
based BIM platform and XR projects capable of sharing a knowledge process based on new levels of 
interactivity and digital creativity. 


Keywords 
extended reality CXR), artificial intelligence (Al), cloud-based BIM platform, interactivity. 
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Fig. |. Workflow of the 
proposed integrated 
approach: from scan—to— 
BIM to extended reality 
(XR) and cloud system. 


Scan-to-BIM Process Meets Computer Science Development and APIs 


Emerging technologies such as eXtended reality (XR), artificial intelligence (Al) and building 
information modeling (BIM) provide innovative opportunities to increase the transmission 
of tangible and intangible values of heritage buildings during the building life cycle and inte- 
grated design process [Alizadehsalehi 2020]. In the digital cultural heritage (DCH) domain, 
the transmissibility of those values is crucial during the processes aimed at the preservation 
and restoration of buildings. The interdisciplinary nature of these processes requires inter 
active environments capable of sharing a large amount of data in real-time among all the 
professionals involved in the process [Banfi 2020, pp. | 6-33]. 

In recent years, innovative results have been found in various fields of application such as en- 


+ +. 


tertainment, gaming, healthcare, marketing and consumers, retail and education. At the same 


+ 


time, sectors such as architecture, engineering and construction (AEC) industry, arts and de- 


+ 


sign, real estate, tourism, and automotive industry, have benefited from new state-of-the-art 
tools (software and hardware) capable of increasing the levels of interactivity, immersion and 
nowledge of digital models [Banfi, Oreni 2020, pp. | 1-1 36; Brumana et al. 2020, pp. 391-400]. 
Accordingly, the challenge to create new and increasingly innovative solutions able to support 
experts in DCH domain has been taken up by various institutions, research centres and Eu- 
ropean projects [Alizadehsalehi 2020]. For all those reasons, the aim of this research was to 
create a digital repository capable of enhancing the use of informative models deriving from a 
scan-to-BIM process capable of effectively representing the reality detected through which is 
possible to share a huge amount of building information (fig. |). 

Scan-to-BIM process, Al and cloud system have been extensively investigated in various 
forms, methods and projects in the last decade [Giordano et al. 2018, pp. 50-73; Graham, 
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Fig. 2. The developed 
cloud — based BIM 
platform and the display 
of different type of 
projects. 


Chow, Fai 2019, pp. 553-568; loannides, Magnenat-Thalmann, Papagiannakis 2017]. Several 
types of research have shown how the integration of those disciplines allows professionals 
to lay the best possible bases for all those types of BIM — based analysis that require high 
levels of detail (LOD) and information (LOI) such as structural analysis, building monitoring, 
infrastructures (InfraBIM), energy analysis, advanced prototyping, plant engineering, sched- 
uled maintenance, construction site and restoration [Luigini et al. 2018, pp. 288-302; Oreni 
et al. 2017, pp. 153-158; Porro, Cocchiarella 2019, pp. 40-56]. Furthermore, several cloud 
solutions have been proposed by the main software developers such as Autodesk, Graphi- 
soft and Bentley to host a large quantity of data (models and information) in a shared form. 
Besides, online applications such as Autodesk A360, BimX have demonstrated how through 
new exchange formats (proprietary and open-source) it is possible to share the wealth 
included in digital models via a web interface within everyone' reach. The next challenge, 
therefore, was to share geometry and the contents in a single solution and through different 
types of devices (PC, mobile phone, tablet and VR headset) e 3D exchange formats (pro- 
prietary and open-source), supporting different types of users in their own daily practices 
and operations in real-time (fig. 2). 

The research and development activity conducted in the last two years by the author focused 
on reaching new forms of interaction, immersion and sharing through the creation of an open— 
source cloud BIM-based platform capable of sharing different types of data such as materials, 
building techniques, monitoring data, schedules/quantities and digital drawings. Thanks to definition 
of novel Grades of Generation (GOG) based on Al,pointcloud segmentation and automation in 
NURBS-to-BIM modelling has been possible to digitalized the reality with high LODs and accu- 
racy [Banfi 2020, pp. | 6-33]. The next paragraph describes the research-development process 
that led to the creation of this platform, focusing on the development and sharing of the main 
XR and HBIM projects developed in recent years. In particular the Basilica of Collemaggio in L 
Aquila, the Basilica of Sant'Ambrogio and the Azzone Visconti bridge in Lecco, represent the main 
case studies developed and included in this digital library, but at the same time, they represent a 
starting point for an open development logic and new levels of interoperability and sharing ready 
to welcome other experiences and architectural artifacts of high historical and cultural value. 


Model Interoperability: Cloud—Based BIM Platform and Extended Reality (XR) Projects 


Tests and analyzes conducted on the new paradigm of interaction between the real and virtual 
world and thanks to the implementation of new scan-to-BIM requirements (GOGs based on 
Al, pointcloud segmentation and automation in NURBS-to-BIM modelling), computer languag- 
es and application programming interfaces (APIs), it was possible to create an interactive data 
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Fig. 3. From Scan-to- 
BIM and XR projects to 
AR library and cloud 
based BIM platform for 
multiple devices. 


repository composed of case studies of national and international interest, integrating various 
ways of real-time visualization of digital models. In particular thanks to the Forge Platform and 
Revit APIs doc which offer Web-based, searchable, and extensible API documentation, it is pos- 
sible to undertake IT development process oriented to the creation of custom digital hub for 
different research case studies. Design and building a viewer that converts and displays models 
on a browser favoured high level of information sharing, moving from professional and propri- 
etary platforms/formats to an open-source logig. As shown in figure 2, BIM modeles (as-found, 
as—designed and as—built projects), once uploaded to cloud using the IFC (Industry Foundation 
Classes) format, can be explored, modified, commented and analysed. Furthermore the devel- 
oped platform allow the upload and download of a many varieties of data type such as Viewable 
2D and 3D design file formats (3DM, 3DS, ASM, CAM360, CATPART, CATPRODUCT, CGR, 
DAE, DLV3, DWF, DWFX, DWG, DWT, EXP F3D, FBX, GBXML, IAM, IDW, IFC, IGE, IGES, IGS, 
IPT, JT, MODEL, NEU, NWC, NWD, OBJ, PRT, RVT, SAB, SAT, SESSION, SKP SLDASM, SLDPRT, 
SMB, SMT, STE, STEP STL, STLA, STLB, STR WIRE, X_B, X_T, XAS, XPR), media and office file 
formats (AVI, GIF, JPG, PNG, TIFF DOC, PDF, PPT, TXT, XLS,...). It should be emphasized that the 
export of the BIM project in the open IFC format and the subsequent upload of the file to cloud 
platform has avoided the loss of the information previously included in the BIM model, favouring 
a easily 3D / 2D read by expert and non-expert users. Thanks to a simplified interface, user can 
select each BIM objects and read all the parameters and information previously inserted and 
connected to the BIM project. This last phase has allowed one to move from a digital logic based 
on proprietary files to an open common data environment (CDE) and real-time sharing. Finally, 
thanks to the level of interoperability achieved the final development step was to include the 
ability to share VR and AR projects (developed using Unreal Engine ad its visual scripting Blue 
Prints) via multiple devices and HBIM models at the same time. Figure 3 shows the multiple 
configuration and devices which could be used for dynamic access to data and digital models 
enabling them to immersive projects that expand our real-world and combine it with virtual 
elements and contents. 


Research Results and Future Prospectives 


The cloud BIM—based platform developed allows the sharing of BIM projects, the remote 
immersion in eXtended reality (XR) projects and the implementation of an augmented 
reality (AR) library of architectural, artistic, historical elements cultural and unique of their 
kind. The main results of this research development process are: 


XR development based on Visual Scripting 
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e creation of a cloud platform capable of hosting and sharing scan-to-BIM projects char 
acterized by the use of a large quantity of data such as point clouds coming from laser 
scanning and digital photogrammetry (primary data sources) and historical reports, 
digital drawings, multimedia files (secondary data sources); 

e greater coordination and collaboration, efficient workflows, user friendly 3D visualiza- 
tions and better project results 

e making VR projects more accessible by sharing executable files for the installation of 
dedicated apps for each project developed; 

e implementation and sharing of AR objects, 

e improvement of the levels of interoperability of digital models through the use of ex- 
change formats (proprietary and open-source). 

e diversified modes of use through smart glasses VR headsets, PCs, mobile phones and tablets. 

Future developments are oriented to increase the level of interactivity of the cloud platform 

thanks to the connection of data coming from monitoring. The final goal will be to integrate 

digital models and real-time data into one digital solution, helping to increase the awareness 
and intangible values of our built heritage. 
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REPRESENTATION CHALLENGES 
Augmented Reality and Artificial Intelligence in 
Cultural Heritage and Innovative Design Domain 


H-Bim to Virtual Reality: a New Tool 
for Historical Heritage 


Carlo Biagini 
Ylenia Ricci 
Irene Villoresi 


Abstract 


In recent years, the application of Building Information Modelling to cultural heritage has led to the de- 
velopment of solid operating methods that have enabled a more efficient management of information. 
The use of other real time methodologies, such as Virtual Reality (VR), has also begun to be tested in 
the architectural context. The objective of this work is to test whether BIM models can be exploited 
to create immersive experiences in digitally simulated environments, with the aim of setting up new 
visualization and evaluation modalities for the built space. Giovanni Michelucci's Church provides an 
opportunity to test the use of Historic-BIM (H-BIM) models for the development of VR. 
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H-BIM, virtual reality, revit, unreal, Giovanni Michelucci. 
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Fig. |. Left: 3D model 
of the church in Revit. 
Right: the same model 
imported in Unreal. 


Background 


The use of Virtual Reality in the architectural field has recently become more and more 
widespread, opening up new opportunities for interaction between people and the built 
environment. In the case of historic buildings, it is becoming a powerful tool for the pres- 
ervation and the enhancement of the historical heritage. This might also give to the profes- 
sionals involved in the conservation process the chance to exploit it for decision making. 
For many historic buildings, most of the available documentation consists of the original pa- 
per documents dating back to the time of construction. Lacking a pre-existing digital model, 
one often needs to be created specifically for VR applications. However, the increasing use 
of H-BIM has made digital models of these buildings available in some cases. To what ex- 
tent BIM models can be re-purposed for VR use is, however, unclear, and this will be the 
objective of this work. The subject of the current study is the Beata Maria Vergine Church 
designed by Giovanni Michelucci. The church was built in 1957 for the industrial village of 
Larderello, near Pisa in Tuscany, Italy. Most of the original documentation has been provided 
by Fondazione Giovanni Michelucci, Florence, Italy. 


BIM-VR Interfacing 


The BIM model for the church was built using Autodesk Revit. Unreal Engine is the game en- 
gine chosen for the implementation of the VR. The communication between Revit and Un- 
real is managed through Datasmith, a collection of tools and plug-ins that enables models, 
scenes and layouts built with a variety of 3D design applications to be imported into Unreal. 
With the ‘Export Datasmith’ Plug-in, elements from Revit are exported as ‘udatasmiht' files. 
Through the Datasmith importer; the content of these files is transformed into a set of “‘ac- 
tors” (the objects of a scene). Datasmith enables the export of geometries, materials and 
textures and their assignment (both Autodesk default materials and customised ones), lights 
and cameras. Visibility settings need to be carefully selected in Revit prior to export, since 
Datasmith will only export objects set as visible in the Revit 3D view. 

Every solid geometry inside a Revit project is imported into Unreal as a Static Mesh Actor A 
Static Mesh is a piece of geometry made of polygonal faces connected to create the object. 
Therefore, the Static Mesh Actors are created as a collection of surface faces, not tied to a 
unique solid volume. Every Static Mesh Actor in Unreal is generated based on families de- 
fined inside Revit. If there is more than one solid inside a family, these parts will be imported 
as a single actor, and cannot be selected separately in the main environment. 
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Fig. 2. Left: wall material 
assignment using Paint 
tool in Revit. Right: same 
wall material assignment 
after import process in 
Unreal. 


During the import process, some identification codes and tags from Revit are imported 
along with each geometry: |) the name of the family; 2) two identification numbers auto- 
matically assigned to the object by Revit; 3) the level the object was linked to in Revit, and, 
if present, the identification number of the hosting element. 

The identification data match between environments, maintaining the correspondence be- 
tween the objects in Revit and the actors in Unreal. This makes it possible to carry out 
targeted substitutions of elements from Revit in Unreal. 

Datasmith also imports materials and textures from Revit as Unreal assets. The static mesh 
actors maintain the material and texture assignment from Revit families (fig. |), but with 
some limitations. For example, every face of the geometries is mapped, meaning that 2D 
coordinates are assigned to the vertexes of every surface. This coordinate system is used 
to define how the texture of the materials is applied to the faces of the solid geometry. In 
Revit every surface of a 3D object has the same default coordinate system, so that textures 
are applied with the same orientation on every surface. However, Revit does not allow a 
flexible coordinate system to manage the orientation of textures assigned to the materials. 
This might result in unrealistic texture visualisation inVR.When this occurs, it is necessary to 
import the element into another 3D design application where it is possible to modify the 
texture mapping data. Then, the element can be imported again into Unreal, and the faces 
of the mesh will show the texture with the proper orientation. 
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Another issue about the import process regards the loss of information caused by the use 
of the Paint tool in Revit. In Revit, the material of the external faces of a volume can be 
modified separately from the main material assigned to the volume of that same object. 
This change, applied using the Paint tool, only affects the external appearance and is not 
transferred to the data about the main material of the volume. Individual sub—regions of a 
surface can also be modified with the Paint tool. However, the export procedure assigns 
only one material to the entire object. Also, the information about the shapes of existing 
sub-regions is not transferred during the export to Unreal. For example, in our case study, 
some sub—regions have been created to add pieces of decorations on some of the walls 
and floors. In the example reported here (fig. 2), the decorations were created as square 
patches on the surface of the wall. Once imported in Unreal, this surface information was 
completely lost: the patches disappeared, and the appearance of the entire wall matched 
the material assigned to the decorative patches in Revit. 
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Management of the VR Model 


As previously mentioned, information about materials and textures is maintained during the 
transfer process. However, the final result in Unreal is rather rough and incomplete, probably 
due to the different encoding of material properties between the two applications. In Revit, 
materials are managed through a pane where the appearance properties are listed together 
ith their assigned values. This list is fixed based on the class of the material and it is not 
possible to add or delete them. Properties can only be changed by modifying the values of 
their parameters. 
n Unreal, materials are one of the most critical and complex aspects to handle, because 
of the wide variety of options for material characterisation managed through a network of 
visual scripting nodes (called Material Expressions). Every node contains a piece of code 
that is responsible for one aspect of the physical behaviour of the material. Different nodes 
are connected to achieve optimal visual results, especially when it comes to the interaction 
of surfaces with light. 

However, in Revit, some material properties are either not modelled or managed in back- 
ground without being explicitly reported in the data for the material. Instead, Unreal needs 
specific handling of these properties to achieve optimal visualisation. Nevertheless, during 
the import process, the Material Expressions are automatically populated with nodes, many 
of which are not necessary for the specific material or contain incorrect values. Therefore, 
these Revit-derived material structures might be very difficult to interpret and handle in the 
Material Editor. In our specific case, we often resorted to re-programming the network of 
nodes entirely, retaining only the properties relevant for the visual restitution in VR. 
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Addition and Display of Informative Resources 


One of the most important features of BIM models is that all kind of information can be 
included directly inside the project. Unfortunately, this information is not exported in Unreal 
and cannot be automatically recalled in VR. In this case study, we tested the use of info-points 
as a method to include some pieces of information manually for real time visualisation of the 
resources. This was limited to a small portion of the information, but the same methodology 
can be extended to the rest of the data. The info-points are specific spots inside the scene 
where it is possible to bring up multimedia panels presenting the information. These panels 
consist of screens, realized with a blank plane rectangular actor, showing manually inserted 
media content, both in the form of videos and images. 
These panels are normally hidden and can be made visible through QR codes linked to 
them. These QR codes mark the location of the info—points and can be activated by the 
user with the controllers by proximity. 
When the QR code is activated, the panel pops up displaying its content. The info—points 
were placed in meaningful locations inside the scene, next to points of interest, for which 
they provide information when queried. For example, a panel describing a stained-glass 
window was located just below this object. The user could visualise this additional material 
while standing in front of the window in the VR environment. 
However, when it comes to generic information, such as blueprints, this methodology 
showed some limitations, because the panels could only be recalled at the specific locations 


where the info-point had been placed. Therefore, optimal solutions will need to be devel- 
oped for this kind of data. 


Conclusions 


Using BIM models as a base for building VR is possible and it is effective in the transfer of 
3D geometries. However, the communication between the two applications used in this 
study is still not optimised for seamless transfer of data. Of notice, these are two stan- 
dard applications used in their respective fields. It is therefore expected that the challenges 
described in this work will be faced by other users and designers interested in BIM-VR 
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Tab. |. Problems met in 
the transition from Revit 
to Unreal and possible 
developements to 
address them. 


either platform allows modification of | Developing a plug-in to add this specific 
surface mapping to adjust texture orien- | feature to Unreal 
tation when applied to the objects 


Material properties assigned with the Paint | Implementing automatic separation of 
tool in Revit are not transferred to Unreal | surfaces with different materials to preser- 
ve the correct assignment in Unreal 


The encoding of material properties is dif- | Optimizing the translation of Revit mate- 
ferent in the two environments rial properties to the visual scripting envi- 
ronment used in Unreal 


Implementing transfer protocols within 
the Datasmith plug-in to allow efficient 


t is not possible for BIM metadata to be 
automatically transferred to Unreal mo- 
dels integration 


interfacing. Hence, it is necessary to streamline the communication between the people 
and teams responsible for these two aspects of project design. It is also important to note 
that BIM models are often insufficient to convey all the necessary information for optimal 
material modelling and rendering for VR applications. Therefore, external references are of 
paramount importance for the correct modelling of the materials in VR. Finally, the passage 
of metadata between the two modelling environments is quite limited and the rich BIM in- 
formation often has to be manually transferred in the VR model. This could further increase 
the workload in case of revisions and updates. The table below summirizes the bottlenecks 
in the process, suggesting possible developments to address these Issues. 
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REPRESENTATION CHALLENGES 
Augmented Reality and Artificial Intelligence in 
Cultural Heritage and Innovative Design Domain 


Experimental Value of Representative 
Models in Wooden Constructions 
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Giulia Pelliccia 


Abstract 


Drawing has always been a necessary model for directing architectural realization. The design of the 
shape has become a nodal condition for multiple analysis integration, which is increasingly necessary 
to meet the performance requirements to which the design must react holistically as a result of digital 
representation, which has resulted in the enrichment and sophistication ofthe simulation's predictive 
capacity The aim of the study is to test the theories that have been proposed and to ensure that the 
results are accurate. It is important to test the accuracy of the adopted solutions using the models 
themselves, through the use of an empirical approach that must be abstracted into a constructed 
representation capable of synthesizing the qualities to be detected. The current research, which has 
resulted in the development of some case studies in the field of wooden constructions, is set in a 
framework that emphasizes the relationship between simulation and realization. 


Keywords 
multi-objective optimization, digital simulation, wooden structures, generative algorithms, parametric design. 
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Introduction 


The importance of representation as a model-building site is focused on the digitization 
process and the convergence of the different aspects of the form into digital computational 
tools. Models gather and analyze data and information through interconnected and interdis- 
ciplinary routes in order to turn them into knowledge. Due to its transdisciplinary nature, 
representation becomes the language of knowledge incorporation, introducing its own field 
of experimental and heuristic intervention, with paths that must be validated. 
The research, which was formed as part of a collaboration between the Department of Civil 
and Environmental Engineering and Abitare+, a local creative wood construction start-up, 
provided an opportunity to put in place a direction that aimed to achieve these objectives 
of innovation and knowledge transfer Simultaneously, joint competencies have been de- 
veloped to kick-start product and service innovation processes. The analysis of generative 
models aimed at multi-optimizing the form, energy efficiency, structure, and cost of wooden 
houses leads to the integration with BIM models. To encourage this practice, an Ames room 
has been developed with a high media impact to highlight the proposed creative approach. 
The study then moved on to multi-optimization of wooden structural walls from this first 
direction. Simultaneously, a study of a new ‘breathing house’ model was conducted, applying 
responsive solutions to indoor hygrometric changes. To check the validity of such solutions, 
a test room was built as an abstract representation of a wooden house reduced to the size 
of a paradigmatic room. 
Generative modeling, BIM, and software solutions unique to the various disciplines involved 
are useful tools to integrate architectural, representative, positive, resources, and communi- 
cation aspects. Once ready for the industrialization process, the models, which in this case 
are materialized in a physical form (the model of the model), must be tested. 


The Research Path 


The analysis of generative models [Bianconi, Filippucci, Buffi 2017; Filippucci 2012] is fol- 
lowed by a proposal for an integrated mass customization—based design and production 
process [Duarte 2005; Paoletti 2017], which is aimed primarily at wood construction tech- 
nicians and specialists but also useful as a dissemination tool for students and researchers. 
The study looks into the possibility of using generative models and evolutionary principles 
to inform the design and customization process. The first case study [Bianconi, Filippucci 
2019] aims to provide individualized housing designs to central Italy. 

These square—plan houses, designed as modular solutions that can be transported and as- 
sembled easily following a simple manufacturing process, can be combined to build custom 
multi-family homes and villages that conform to both the environment and their inhabitants. 
The generative process specifies a variety of design options, all of which depend on genetic 
algorithms to adapt and optimize the architectural model. The design concept is focused on 
the analysis of local codes and X-Lam and Platform-Frame building systems with the goal 
of reducing waste and optimizing the construction process. With the study's goal in mind, 
energy consumption, thermal, and visual comfort, as well as price, were evaluated with the 
construction company and through iterative processes. The results of this first study, which 
began with the selection of solutions available to the company, have prompted a closer 
examination of each element that makes up the envelope. 
The realization of the Ames room is exemplary in this regard [Ames, Ittelson 1952]. It is 
an application of perceptual theories [Arnheim 1965; Gibson 2018] that reviews digital al- 
gorithms to concretize an architectural expedient based on image culture [Pinotti, Somaini 
Elcograf 2016] and that has led to a model that synthesizes the research's multiple questions 
[Bianconi, Filippucci 2020]. 

The focus of the investigation then shifted to improving the energy efficiency of wooden 
structures that had previously been customized to meet the location's specific require- 
ments. The aim in this case is to use generative design tools to optimize the preliminary 
cost and efficiency of wood walls for X-Lam and Platform-Frame structures, with the goal 
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Fig. |. The selection 

of various possible 
optimized walls according 
to thermal parameters, 
materials or cost. 


of evaluating the actual performance of the built solutions [Seccaroni, Pelliccia 2019]. As a 
result, the described workflow begins with the implementation of parametric algorithms 
in Grasshopper that return thermal transmittance, decrement factor time shift, costs, and 
verify the absence of interstitial condensation while varying the wall materials and thick- 
nesses from time to time [Aste et al. 2015; Rossi, Rocco 2014]. The selected parameters 
can be processed in a multi-optimization direction based on the application of evolution- 
ary principles through the Grasshopper plug—in Octopus [Diakaki, Grigoroudis, Kolokotsa 
2008], in which more than 5000 possible material and thicknesses combinations have been 
automatically analyzed. 
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The best solutions can thus be selected, identifying the Pareto front [Wang, Zmeureanu, 
Rivard 2005; Wright et al. 2002], in which the combinations simultaneously present optimal 
values of the various parameters that evaluate the walls summer and winter behavior, as 
well as the overall cost (fig. |). Because of Octopus’ genetic multi-optimization, the ap- 
proach just mentioned is based on the development of an algorithm capable of simulating 
human decisions in finding the most suitable solutions from both an energy and economic 
standpoint. As a consequence, the algorithm acts as an Artificial Intelligence (Al) [Ridolfi, 
Saberi 2019] because it simulates human decision-making and also allows for the testing of 
a wide number of potential combinations. The research then moved from virtual to phys- 
ical with the construction of a test space. The test room is an abstract representation of a 
wooden house scaled down to a paradigmatic space's dimension. It stands out thanks to a 
removable wall that can be replaced with any kind of X-Lam or Platform—Frame construc- 
tion. This architectural aspect was realized with one of the walls optimized by the algorithm 
and was monitored once fitted with sensors and data acquisition systems to compare the 
data obtained from simulations with the real ones and thus understand the actual usefulness 
of the optimization tools used. The in situ transmittance was calculated using thermocou- 
ples, flux meters, and additional temperature and humidity probes, and compared to the 
algorithm simulation using UNI ISO 9869 [ISO 9869—1:2014 Thermal Insulation — Building 
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Fig. 2. Thermocouples, 
fluxmeters, temperature 
and humidity probes and 
anemometers inside the 
test room. 


Elements — In-Situ Measurement of Thermal Resistance and Thermal Transmittance — Part 
|: Heat Flow Meter Method 2014]. The obtained results confirm the simulated model's 
consistency with the structure's actual behavior, taking into account a percentage of error 
due to various factors that may affect field measurements (fig. 2). 


Conclusions 


The relationship between digital and wooden constructions opens up impressive fields 
of use, as shown by the integrated action promoted by an international call [Bianconi, 
Filippucci 2019b]. It has also led to the development of a research network involving 
more than 150 researchers from all continents [Bianconi, Filippucci 2019a]. The great 
cultural value of this initiative can be ascribed to the new key role of representation for 
contemporary research. The research outlined the value of preliminary digital simulation 
for form-finding, both in terms of the project's actual final configuration and in terms of 
the less tangible aspects of the building's efficiency As a result, representation takes on a 
new position as the ‘place’ of the model. The dynamic passage between real and virtual 
in a spatial model helps in representing intangibly, with high reliability, what is concretely 
abstract. This demonstrates the representative models’ experimental importance. By using 
artificial intelligence’s analytical capabilities and reinterpreting the flow of data collected 
during the monitoring, the study aims to define and validate the best performing solutions 
for the particular architectural project. 
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Abstract 


In this paper is presented a possible methodology for automation through the use of deep learning of 
BIM modeling starting from different types of formats, such as digital processing of paper documents 
and CAD formats. The work is configured as a proof of concept of a possible contribution that a 
technique currently scarcely used in the architectural field such as deep learning can bring to the 
design, in particular in the realization of the information model, which today represents one of the 
most consuming-time activities. 
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notBIM to BIM 


The production of information, in the construction sector and in a very widespread domain, 


+ 


is undergoing a transformation process in recent years that involves the transition from 
the use of CAD to the implementation of BIM as regards the generation and extraction 
of graphic and non-graphic data. This process, although strongly favored by national legis- 
lation, is still in a phase in which CAD software often plays a role of primary importance, 


especially in the initial stages of design, wh 


ile BIM is sometimes relegated to the final stages 


of the design process, during the final and executive project, only to be abandoned in the 
construction phase. It is also true for many realities that a large part of the built heritage 


preserves documentation in paper or CA 


D format. 


In the production of projects and graphics, the current state of a BIM model shows draw- 


ings drawn up in different formats, with 
disciplines: there are cases in which there 


different methods and purposes for the design 
is a need to create a model for the management 


of an existing building artifact, to draw up a building restoration project, for the variation 
of methodology that can take place between levels of subsequent technical study in the 


design, such as the transition from a final p 
developed in BIM. 
The creation of a BIM model finds in the 


roject developed in CAD to an executive project 


modeling of geometric data one of the activities 


of greatest time consumption, with consequent dilatations in the planning times, which are 


reflected in an increase in costs for organi 
data created with paper or CAD formats, 
only partially exploits the amount of time 
quent waste of the work carried out previ 


from handwritten documents. In addition 


[Koutamanis, Mitossi 1992]. 


+ 


+ 


to demonstrate the correctness ofth 


+ 


plans 


+ 


to architectural representation for 


+ 


through visual programming, Computer Vis 


Methodology 


recognition [Mokhtarian, Abbasi 2004]. These first specialized works, which served as cadastra 


the transition from 2D drawings to three-dimensiona 
models [Lewis, Sequin 1998]. The analysis models have become more specialized, in a direc- 
tion of increasing effectiveness towards different disciplines and electronic media [Dosch et al. 
2000; Lu et al. 2005;Yin, Wonka and Razdan 2009]. The application to the generation of BIM 
models as output has instead been explored only recently in which the recognition is applied 


zations [Deutsch 201 |]. The existence of graphic 
although used as an initial data for BIM modeling, 


spent in the production of such data, with conse- 


ously, as the data starting graph must be manually 


recreated within the BIM model, like for example the production of digital text files starting 


+ 


to being extremely laborious, a data recreation 


process of this type also depends on the experience and expertise ofthe operator involved 


n this work we try to obtain through Deep Learning methodology those geometric data, 
present in paper or CAD drawings, useful for the creation of BIM objects, and to use such data 
for the automatic creation of such objects through algorithms. With reference to the existing 
bibliography, it is a question of updating experiments already carried out with advanced algo- 
rithms, evaluating the speed of execution, accuracy and integrability. It is possible to find basic 
algorithms starting from the early 90's [Kaneko 1992]. Such algorithms mainly work on symbo 


e awards, were followed by applications dedicated 


+ 


to all building elements, such as doors and windows [Lim, Janssen and Stouffs 2018]. In fact, 
this work aims to show how tt is possible to integrate, in a single BIM modeling environment, 


ion and Deep Learning libraries. 


The project exploits existing libraries for 


+ 


todesk Dynamo in its version DynamoCo 


puter Vision and Machine Learning. 
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Deep Learning and Computer Vision in order to 


recognize objects from suitably digitized paper formats and to recreate these objects in a 
BIM environment using VPL. For this purpose, the visual programming environment Au- 


reRuntime 2.7.0 was used, given the impossibility 


of integrating Python libraries for Deep Learning in the current versions integrated into the 
BIM modeling software Autodesk Revit (Autodesk Dynamo Revit 2.5.0 for Autodesk Revit 
202 1), and OpenCV (Open Source Computer Vision Library), a software library for Com- 


The graphic drawings taken into consideration are the plan views, as they represent in most 
cases the most important documents in the architectural field [Koutamanis, Mitossi 1993]. 
The process can be divided into 4 stages: 

e preparation of the image; 

e Image processing; 

e subdivision of detected objects: 

e creation of objects from extracted geometric information. 
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the authors. 


The image preparation phase involves the digitization of images from paper format or 
the transformation of vector files into matrix form. The search for the most suitable 
method for the CAD-matrix transformation is currently underway, however the ex- 
port of tables in JPEG form seems to be the quickest and easiest method for preparing 
the CAD data. 

In the case of digitized paper documents, the creation of tracing elements of the be- 
ginning and end of the analysis area and identification of the coordinates, as well as 
identifying the scale of the objects contained in the document, is deemed necessary. 
In the processing phase, the image imported into the Visual Programming Language 
(VPL) is provided as input to a Python script that uses openCV in order to identify sim- 
ple geometric shapes formed by 4 vertices in the image. The library uses appropriate 
filters to identify, through matrix operations, the presence in the image of vertical lines, 
horizontal lines or edges in order to determine the presence of vertices in a certain 
region of the image. 
The currently implemented algorithm applies a single transformation to grayscale im- 
ages that provides for the application of a threshold to each pixel, for which the pixel 
values above the threshold are brought to 0 while the higher values are taken to the 
maximum value supplied as input, in this case 255.A second function identifies possible 
outlines of elements in the image, which are subsequently discretized by obtaining the 
vertices of the figure. 

The list of vertices is filtered by selecting only the figures whose number of vertices is 
equal to 4. The vertices are then decomposed into their own X-Y components and sup- 
plied as output. 
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Fig. 2. Algorithm operation 
diagram. Elaboration by 
the authors. 
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The list of coordinates obtained is then used to create the related points in theVPL environ- 
ment, which will create four-sided polygons that are analyzed in their geometric information 
in order to perform a further skimming on the objects in the initial image. 
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Fig, 3. Results classification. 


Elaboration by the 
authors. 


Fig, 4. Information 
extraction. Elaboration by 
the authors. 
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The objects are created starting from the geometric or textual information obtained from 
the drawing. In the case of textual information, an algorithm can be set up which, using 
openCV, recognizes text elements, obtains their position and content and associates them 
with the forms detected using the algorithm described above. 
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The currently developed algorithm is an extremely simplified resolution of the identification 
problem and represents a proof of concept of the possibility of integrating this type of op- 
erations in a single environment. 

For a real application, the algorithm requires further developments such as: 

e apart of better pre-processing of the image, in order to ensure better results for non— 
standardized images from the point of view of color, contrast; 

e the training of a neural network that can autonomously classify elements within the 
image based on pre-training on typical representations of architectural elements in plan 
(walls, pillars, floors, doors, windows ...). 

From this perspective, the algorithm would not be limited to the recognition of rectangular 

shapes only, but would be aimed at the recognition of elements of any shape and their clas- 

sification on the basis of the classes provided during the training phase. The algorithm would 
then obtain, depending on the class, different information for the creation of objects in the 

BIM environment. For example, for the creation of wall elements, the axis of the wall (major 

axis of the rectangle) and the thickness (minor axis) would be derived from the identified 

rectangle, for the creation of door elements, the centroid of the identified polygon (position 
of the door in the closest identified wall element. In this sense, we would approach an al- 
most complete recognition of the graphic information contained in the original work, which 

can be divided into three types [Koutamanis, Mitossi 1993]; 

e Recognition of the geometric element: extraction of geometric information by element 
from the graphic; 

e Recognition of the Building Element: extraction of classes by element from the graphic; 

e Recognition of the spatial articulation, training a neural network to recognize the spaces 
formed by the association of different elements (concerning the architectural discipline), 
or zones (concerning the MEP discipline). 


Results 


The results obtained in the processing of an image representing part of an HVAC system 
are shown below. Note how an extremely simple form such as that of the rectangle takes 
on a multitude of meanings in architectural language based on the context in which this 
graphic symbol is placed. The variation in meaning occurs both in relation to the discipline 
and in the relationship between the element and the context. A rectangle can therefore 
represent a wall, a floor or a piece of furniture in the architectural discipline, a pillar or a 
beam in the structural discipline, a rectangular or circular duct or a connection in aeraulic 
systems. The need to train a neural network for the discernment of such cases is there- 
fore recognized. 
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Fig, 5. Information 
extraction. Elaboration by 
the authors. 


It is possible to note that a sketched and very simplified version of the algorithm already achieves 
good results in recognizing simple elements and tracing their axis. In this case, a subsequent algo- 
rithm will be responsible for the grouping of shapes on the basis of the connection by means of 
objects of different shapes, representing in this case the fittings of the ducts.At this point you have 
all the geometric information necessary for the creation of the elements in the BIM environment. 
The work carried out has undoubtedly demonstrated the greater speed of execution of a deep 
leaming/object recognition algorithm compared to methods of analysis according to geometric 
formulas, previously experimented by the authors. To this must be added the high accuracy rate 
in object recognition for simple geometric shapes or scarcely variable shapes achieved by the 
current available algorithms. However; it is necessary to create more complex neural networks 
that can recognize and classify a multitude of objects belonging to the common architectural 
language for a complete or semi-complete automated modeling of BIM objects starting from 
the considered formats. In order to have a fully operational IT product, it is also necessary to 
develop a method for verifying the dimensional correctness of the outpute and the subsequent 
correctness of positioning of the elements in space. 
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Abstract 


The constructi 
spatial process learning 


and how data 


on industry in the digital era has seen significant changes in design, construction, and 


through new technological systems. These processes influence management 
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Introduction 
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The building is a monumental complex characterized by complex architectural elements 
(i.e. vaults) that have changed over the years. These characteristic elements are digitized to 
simulate innovative spatial scenarios through the BIM methodology, making comfortable the 
environments intended for patients’ motor rehabilitation through VR technologies. In this 
case, Unity 3D was selected as the VR platform, while the HTC Vive Head Mounted Display 
(HMD) was selected as the immersive technology. In particular, two specific areas (e.g., gym, 
hospital room) have been identified as the basis for the virtual environments. The immersive 
application has been characterized by aspects related to both the gamification approach 
and the therapeutic elements. 

The second use case selected is the Trompone complex (Moncrivello, VC), which offers 
several healthcare services to support high complexity patients and relatives. lt consists of 
several buildings, including a nursing home, an assisted healthcare residence (RSA), a con- 
vent (part of which is now used as a training center), and the sixteenth—century sanctuary. 
In this case, the attention is given to designing a small daily center for Alzheimer’s patients 
within the existing complex. Starting from a BIM model, different VR software platforms 
were investigated to improve the new spaces’ design, controlling the possible interferences 
between the architectural, structural, and systems disciplines. The interaction with the digital 
model has taken place through two VR tools, Enscape and Autodesk Live. While the first 
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Fig. 3. Construction and 
Participatory Digital 
Twin for AEC Industry 
(authors' elaboration). 
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intelligence through algorithms and augmented reality may soon facilitate their interaction 
through collaboration and interoperability. Such complex systems can facilitate relationships 
between public and private administrators, professionals, and citizens within a new city model 
defined as augmented building/city. The advantage expressed by models with a high capacity 
of response and data cataloging is related to representing the urban/architectural model. The 
transformation of virtual models into construction and participation digital twin allows virtual 
machines to process traditional forms and properties into future elements. 
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Survey and BIM for Energy Upgrading. 
Two Case Study 
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Abstract 


This research aims to deal with the theme of the survey applied to the BIM methodology in cultural 
heritage; in particular on those buildings that, although from different periods and construction tech- 
nologies, require energy requalification. The study focuses on two cases in Umbria: Borgo Lizori, a 
characteristic historical settlement located along the Assisi-Spoleto olive grove, and the Palazzetto 
dello Sport in Bastia Umbra, a post-modern building designed by arch. Leoncilli Massi. 


Keywords 
cultural heritage, energy efficiency, BIM. 


doi.org/10.3280/0a-686.59 


37 | 


Fig. 1.A global view of 
Lizori Burg. 


Introduction 


In recent years there has been a steady increase in demand for the rehabilitation of the 
existing building stock, both historic and more recent. Intervention on the built heritage has 
to face increasing challenges and opportunities, such as the recent tax reliefs issued by the 
government; the resulting expectations are related to sustainability, energy saving, as well as 
the availability of new materials and continuous technological innovations. In this context, it 
is essential to understand the building, tts genesis and its technological characteristics. Par 
ticular attention must be paid during the survey phase which, in the most complex cases, is 
facilitated by the use of drones or laser scanners, which make it possible to obtain a digital 
clone in a short time but with high accuracy. The point cloud obtained in this way is then 
imported into the BIM environment; from here it is possible to reconstruct the geometric 
elements, classified into hierarchical structures, and explored with different levels of detail. 
This procedure was carried out in the two cases that will be presented here; they are two 
profoundly different cases, but from which emerges as the lowest common denominator 
the need for energy regeneration, the point of convergence of a methodological research 
path that starts from the survey and projects itself to integrate the multiple information in 
the BIM environment. 


The Case of Borgo Lizori 


Lizori Burg, a settlement located on the hill between Foligno and Spoleto in the city of Cam- 
pello sul Clitunno (PG) and a UNESCO World Heritage Site, represents the first case study. 
The very first inhabitants of the area were the Umbrian, an Italic people from Central and 
Eastern Europe who came to the peninsula around 1000 BC. Subsequently, the place will take 
the name of the castle of Pissignano, derived from the ancient Pissinianum, or swimming pool 
of Giano. The first nucleus developed in Roman times along the Flaminia, while later the hilly 
one was formed, where a small Benedictine community was established, presumably around 
the eleventh century Between the eleventh and twelfth centuries, the monks erected a wall 
around the inhabited center; which therefore took the name of San Benedetto Burg. Over the 
centuries, the burg was often disputed due to its privileged strategic position. The recovery of 
the Burg, which took place from the mid-seventies, on the one hand gave new life to a place 
that was abandoned, on the other it was an opportunity for the application of good practices: 
the local materials were recovered, put in place with traditional construction techniques, so 
much so that they were shown at the lUA World Congress in 2005. 

Its triangular shape has the summit upstream, with towers originally arranged at the comers and 
on the two sides inclined in an intermediate position. The urban layout is quite characteristic: it 
has a compact terraced layout of buildings with parallel lots and a trend that sets itself on con- 


Fig, 2. Revit, different LOD 
levels of the BIM model. 


tour lines, rapidly degrading along the slope. The village is in a perfect state of conservation and 
presents architectural forms in relation to nature, as well as purely artistic decorative elements. 

As preliminary survey, a model was developed through aerial-photo modeling techniques, 
a technique that offers considerable advantages for dense patterns, such as the case study. 
It was possible to survey the whole village, and represent the roof shape in detail using a 
professional EVO 4HSE drone.The flying and the image acquisition phase lasted 8 hours. In 
this phase more than 20 marker were positioned on the ground to facilitate the aero-tri- 
angulation and to limit the cameras orientation error The camera used for this phase is a 
LUMIX DMC-GH4 digital mirrorless with a single lens, on a Panasonic 14-18 mm optic. The 
software used for photo modelling is ContextCapture (Bentley). 

Once concluded the input and marker georeferencing phase, it is possible to move to the aero— 
triangulation phase, where among the options it is possible to set: density of key points, choice of 
methods of similar photos and adjustments of focal distortions. When completed the cameras 
alignment phase, it was the time to produce the dense cloud choosing points’ density, texture 
quality and the mesh parameters. Subsequently, the survey data were merged and geo-oriented 
through the 3Dreshaper software, thanks to which it was possible to extrapolate further data 
such as the land digital model and the built environment shape to hierarchize the point cloud. 


By selecting the most widely used Autodesk BIM Revit software as a reference tool, oper 
ationally it was necessary to index the geo-referred data through the Recap software to 
ensure direct management in the identified digital environment. From the cloud of points 
is possible to move on to the interpretation of the shapes detected through the modelling 
of geometries. This are differently detailed in terms of scalar levels of detail, addressing the 
multiple issues related to the definition in parametric informational environment of singular 
and peculiar elements, such as those met in historical contexts like the case study. 

Using 3DReshaper software, it was possible to export the BIM Revit model in FBX format 
and overlap it to the cloud of points from the laser scan, to calculate the variance, intended 
as the distance between homologue points of the BIM model and the relief. The obtained 
mistake is the sum of different factors and the average values are in the order of 5cm, an 
acceptable value connected to the vernacular character of urban space and the possible 
needs of a representative approximation. 

The use of this platform has allowed the model to add a series of details relating to the 
structure of the individual buildings, ranging from the stratigraphy of the wall facings to 
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that of the roofs, passing through the types of heating system. This implementation of 
data has made it possible to carry out a series of analyzes on the current state of af- 
fairs from an energy point of view, allowing the development of a series of design strat- 
egies for the redevelopment of the individual element and, consequently, of the entire 
burg. In particular, the building envelope was studied, including both the external walls 
and the roof; a project was carried out for the inclusion of home automation, and the 
lighting was studied, including a study of both the internal and external environments. 


The Case of the Sports Hall in Bastia Umbra 


The Sport Hall in Bastia Umbra was designed by the architect Leoncilli Massi in the 1970s 
as a covered market for the S. Lucia area; later, following the granting of funds by CONI, the 
use of the building was changed to sport hall in what is now the Giontella area. 

The building has an atypical truncated pyramid shape, with a rectangular base rather than a 
square one, and a basement that was originally intended to contain boxes for the open-air 
market. But if the pyramid works by gravity, Leoncilli Massi reverses the logic and inserts an 
exoskeleton, a characteristic element of the entire project. The structure recalls the themes of 
post-modern architecture, such as the colours chosen or rather the arch—window; and again, 
the fragmentary nature found in every corner of the building, where stairways, exits and en- 
trances are inserted. Lastly, the misalignment, with the basement creating an 'L'on the east side. 
Almost forty years have passed since Its inauguration on | 6 November |983,andthe Sport Hall, 
although it has been selected by MiBAC as one of the most important works of contemporary 
architecture, presents a number of critical issues due to the technologies used at the time, such 
as infiltrations and thermal bridges,as well as more'recent’ problems such as parkourand graffiti. 
The survey was carried out using a laser scanner with a spherical camera; in this case, 
through the software installed on the mobile device, it is possible to see in real time the 
cloud of points that is gradually forming, as well as the positioning of the instrumentation. 
(fig. 4) The model that is created is geo-referenced. For simplicity’s sake, the operations for 
importing it into a BIM environment, already seen in the previous paragraph, are omitted. 
This led to a series of hypotheses which, on the one hand, resolve the technological diffi- 
culties, aim at energy requalification and, finally, restore the contemporary architecture. In 
particular, solutions were studied to eliminate water infiltration, redo the window and door 
frames and study thermal insulation. Finally, a proposal for a new external skin that does not 
distort the work but at the same time makes it contemporary. 


Fig. 3. The Sport Hall in 
Bastia Umbra. 
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Fig, 4. Positioning of the 
laser scanner and its point 
cloud in the Sport Hall. 


Conclusions 


This contribution aims to compare two case studies, profoundly different, but which have in 
common the need for energy regeneration. Starting from the point cloud, necessary in these 
cases where the survey operations present objective difficulties, a first step was the implemen- 
tation in a BIM environment; the research will be perfected with immersive experiences in an 
AIAR environment, which will allow a further enhancement of the cultural heritage. Through 
the case studies, the research aims to highlight the centrality of a methodological workflow in 
its ability to manage and interpret data, which becomes fundamental for the knowledge and 
enhancement of the richness of our heritage of historical and environmental interest. 
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A Proposal for Masonry Bridge Health 
Assessment Using Al and Semantics 


Raissa Garozzo 


Abstract 


Masonry railway bridges represent a historical built heritage to be preserved. This contribution pro- 
poses a new methodological approach for health assessment of masonry railway bridges based on the 
definition of image-based and Al-driven survey protocols useful for the creation of semi-automated 
H-BIM models. To do this, a heuristic approach is required with the integrated combination of tech- 
niques and methodologies belonging to different fields. As case studies the masonry bridges of the 


sicilian Circumetnea railway are chosen. 


Keywords 


railway masonry arch bridge, digital survey, archival research, artificial intelligence, GANs. 
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Introduction 


Masonry arch bridges are one of the most common structural typologies in the worldwide 
architectural heritage. In the nineteenth century, the design and construction of bridges had 
a great impulse with the railways’ advent. While the function of the bridges has changed over 
time, due to the abandoned railway networks, their value as historical permanence of the 
past has increased over time, even enhancing the value of the landscape. 

The architectural qualities of these assets make them testimonies of past theoretical 
knowledge and construction skills; the natural obsolescence threatens this key heritage due 
to neglect and lack of maintenance in the case of disused bridges. Therefore, a remarkable 
number of bridges have collapsed or failed worldwide over the past years. The stability 
and safety of still-in-use masonry bridges must be guaranteed, avoiding over-dimensioned 
intervention and, in the worst-case scenarios, demolition and reconstruction. 

Hence the need for a strategy of expeditious health monitoring of masonry arch bridges, aimed 
at containing the potential damages caused by seismic, hydrogeological, and other vulnerability 
assessments. It is crucial to develop effective and integrated procedures to characterize the 
structural conditions, identify and prevent potential vulnerabilities of such historic assets 
studying their geometric configurations, construction techniques, and documentary heritage. In 
this direction the current Italian guides for risk classification and management, safety assessment 
and monitoring of existing bridges give pivotal suggestions. 

H-BIM approach could be the starting point for management, conservation, and maintenance 
of these historical assets, as it gathers all present and past information on the artefact. Besides 
that, H-BIM could be used throughout the life cycle of a bridge, as a management supporting 
tool constantly updated. The knowledge base of such an information model is the point cloud, 
acquired with different methodologies depending on the characteristics of the case study. 
Despite the tremendous advantages of this technology, one of the principal limitations is 
the time and training required to manually process the huge quantity of data that point 
clouds contain. In this regard, Artificial Intelligence (Al) techniques and semantic web could 
really help in the process. For example, ontologies could serve in helping neural networks 
in semi-automatic point cloud segmentation or adding semantic layers to H-BIM models. 
Also, the bridge's state of health is commonly assessed by visual inspection, and this is a time 
consuming, expensive, and laborious procedure. 
Computer vision with Deep Learning (DL) techniques can support professional users in damage 
detection and classification. However, Al methods, based on the DL paradigm, require a significant 
amount of annotated data that, in this context, it is often impossible to collect. To make up for 
this issue it is possible to exploit data augmenting techniques to create synthetic data to enrich the 
dataset, through an approach based on Generative Adversarial Networks (GANs). 

In this regard, this manuscript aims at proposing a new methodological approach to define image— 
based and Al-driven survey protocols useful for the creation of semi-automated H-BIM models 
for health assessment of masonry railway bridges. 

To support the research purposes, the chosen case studies are the masonry bridges of 
Circumetnea, a still-in-service railway that almost encircles the Etna Volcano, passing through 
several towns in Etnas foothills. 


State of Art 


Automating the point cloud-to-Bridge Information Models (BIM) process can significantly 
reduce the effort and cost in masonry bridge inspections and management. 

Recently, Zhao and Vela [2019] provides a pipeline for simple concrete bridges scan-to— 
BIM, integrating several procedures related to segmentation, surface model estimation, 
and classification of surface regions. Also Xu and Turkan [2019] underlines the need to 
identify productive approaches to inspect and manage bridges, proposing a framework that 
integrates BIM and Unmanned Aerial System Services (UASs) technologies. In Leön-Robles 
et al. [2019], historical-archival research and digital survey are used as a database to create 
a Historical (H)BIM. 
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The automatic or semi-automatic point cloud segmentation can considerably support 
information modelling. Kim et al. [2020] present a methodology for the automated concrete 
bridge component recognition using deep learning, while Truong-Hong and Lindenbergh 
[2021] introduce an approach to automatically extract the point cloud of each surface of 
structural components of a slab bridge. The researchers are showing an increasing interest 
in an effective pipeline for masonry bridges as well. In Riveiro et al. [2016], a method for the 
automatic segmentation of historic masonry bridges was achieved based on the combination 
of a heuristic approach and image processing tools adapted to voxel structures. Using these 
results, a methodology to automatically transform classified point clouds into IFC models for 
further applications is proposed by Sánchez Rodríguez et al. [2020]. 

Given this, a BIM approach supported by automatic and semi-automatic point cloud 
segmentation can help protect, managing, and enhancing historically rich assets, such as 
masonry arch bridges [Savini et al. 2021]. 


Methodology 


The methodology proposed is structured as it follows: 

Cognitive phase, aimed at knowing and documenting the case studies, was developed 

according to the following steps: 

e Documentary research — it allows the understanding of the design idea and provides 

important indications on the artefact, especially regarding the geometric configuration 

and the relationship with the pre-existing structures at the time of construction; 

e Integrated digital survey campaigns — it is aimed at acquiring the geometrical and material 
configuration of the bridges and is carried out by using TLS, SFM photogrammetry and 
the experimentation and validation of videogrammetry techniques; 

e Census - it brings together all the information acquired during the cognitive phase in a 
geographical information system, allowing to understand the bridges analysed relation 
with the entire railway and the territory. 

Semantic phase, aimed at designing a masonry arched bridge computational ontology that 

will be used as a knowledge-base for deep neural network training, needed of: 

e Conceptualisation — it provides an in-depth survey of existing vocabularies and 
taxonomies. Then, identifying relationships and hierarchies between parts to choose the 
proper classes, subclasses, and property of the developing ontology is required. To do 
this it is crucial to conduct in-depth research into technical manuals and treatises, as it is 
analysing several case studies and their typologies, as reported in. 

e Comparison of existing ontologies — it is useful to understand whether it is better to link 
to something existing or create a new ontology. 

* Ontology development — for which a particular attention is given to the level of granularity, 
if it is as an extension from existing standards. Some levels of information to be added, for 
instance, are related to the semantic structure, construction techniques, and typical defects. 

e Test phase — in which the developed ontology is used for training a deep neural network. 
In doing this, an attempt is made to understand if the chosen ontology standard fits wel 
with the research purposes. 

Artificial intelligence (Al) approach, which aims to train a neural network for point cloud 

segmentation and visual-based defect detection, both on the numerical models and images. 

e Creation of a two-level GAN - it is composed by a first GAN that generates isolated 
objects corresponding to each ontology concept, and a second one that combines the 
generated objects according to the spatial information provided by the ontology, to 
generate realistic scenes. To create a synthetic dataset, it is necessary to start from rea 
images. For this reason, a massive acquisition of pictures of the same typology as the case 
study is necessary. 

e Training of the DNN with real and previously mentioned synthetic datasets 

e Testing phase on existing bridges point clouds 

Information Modelling phase, almed at creating parametric components, starting from the 

segmented point clouds. 
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First Results 


The case study chosen for the proposed experimentation is the Circumetnea railway masonry 
arch bridges. Built between 1889 and 1895, the Circumetnea is a regional narrow-gauge railway 
that connects Catania to Riposto, almost encircling the volcano Etna and passing through several 
towns in the foothills of Etna. The heterogeneity of the typologies (number of arches, materials, 
geometry) and the recurrence of the types, which include many instances, made them the ideal 
case study for this research. Finally, it is a heritage at risk, because some of the bridges have 
recently been demolished because of changes in traffic requirements. 

Cognitive phase — The first step in the cognitive phase was an in—depth documentary research 
carried out at the State Archive of Catania, which holds 192 folders of original drawings and 
documents dated back to the timing of construction of Circumetnea. Plans and longitudinal 
profiles of the rail route, together with bridge projects and metric computations, were 
acquired digitally. This documentation allows the investigation of unknown construction and 
technological features, such as foundation typologies, and a better understanding of the reasons 
and the developments of the project. The documentary research was coupled with the digital 
survey campaign. The integration of several surveying methodologies, such as laser scanning, 
photogrammetry (ground and drone-assisted) and videogrammetry, was required due to 
the peculiarities of the analysis. A total of 37 bridges were identified along the route under 
investigation. Only 13 of these were accessible. 18 of them were photographed and only | | 
were surveyed, using the above mentioned techniques (fig. 1). The census was conducted 
concomitantly with the previous two stages. The bridges were first detected using Google Earth 
and then loaded into QGIS, in order to produce a grid of attributes organised according to the 
position (latitude and longitude), number of arches, building materials, archival documentation 
and survey operations that were carried out. 


Semantic phase — The creation of a specific masonry bridge ontology started with the 
conceptualization phase. At the first stage, a multifaceted bottom-up analysis of the elements 
constituting this architectural typology, referring to well-defined thesauri and taxonomies, such as 
the AAT (Getty Research Institute) was conducted. 

The actual conceptualization phase then began, based on the comprehensive text [Torre 2003], 
which helped in the semantic organisation. The element composing bridges are included in Physical 
Object, that could have Physical Property, such as materials, shape, and defects. Classes, as well as 
materials and shape, were obtained consulting manuals and historical documentation and through 
the case study observation. For defects, some inspection sheets previously produced for the 
Circumetnea under an agreement with our Department were consulted (fig. 2). Here, it moved 
to the comparison among existing ontologies; the analysis was conducted on CIDOC, ISO 21127 
standard, which is the fundamental ontology for the management of cultural heritage information, 
and on IFC, ISO 16739 standard, open format for the management of interoperability. 

Tothe author knowledge, there are currently no CIDOC extensions on masonry bridges. It is, therefore, 
interesting to explore the opportunities on such a diverse asset, which require a multidisciplinary 
approach for the safeguard of our cultural heritage. Nevertheless, the IFC-bridge is currently a project 
under development; also, using a single standard ontology as IFC to manage the semantic of built 
heritage in multiple applications could be an exciting challenge that simplifies information management 
and potentially bridging the gap between digital surveying, information modelling, and Al applications. 
In the light of this, tt was decided to extend the ‘IFC bridge’, acting on its granularity. 
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Fig. 2. The 
conceptualization schema 
(on the left); the detail 
of the defects developed 
part (on the right). 


MATERIAL 
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Physical Object 
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Al approach — At this early stage of the research, the necessary material for the creation of the 
synthetic dataset was acquired. A masonry arch bridge dataset was collected by the web, consisting 
of 10.446 images of 3.000 masonry arch bridges. 

These data were built using a web scraping technique on Structurae.net, an international database 
and gallery of structures. Data collection was a key point in the training process, as a poorly built 
dataset could lead to bad performances or prevent the GAN model from learning. After images 
were scraped it took several days of manual data cleaning, erasing bad quality images, to obtain the 
final dataset used for the training, Images of aqueducts, drawings and plans, images with altered 
colouring, images that did not represent significant elements of bridges (such as statues and, more 
generally, decorative elements) were removed from the dataset. The images left after cleaning 
operations is a total of 7434 images. 


Conclusions and Future Development 


The results obtained at this stage of the research are encouraging, the conceptual formalization 
of the ontology added a new layer of knowledge to this valuable heritage. The experimentation 
will continue focusing on the integration of the ontology and, consequently, on the creation of the 
synthetic dataset and the segmentation of the point clouds. With regard to parametric modelling, 
an in-depth investigation will be carried out to identify a good pipeline to maintain a strong model 
fit with parameterisation of tts components. 
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Al for AEC: Open Data and VPL 
Approach for Urban Seismic Vulnerability 


Federico Mario La Russa 


Abstract 


This contribution provides an overview of the VPL evolution and an application case concerning the 
classification of seismic vulnerability indices with Al. This research aims to contribute to the scientific 
debate on the use of these technologies in architecture, deepening the themes of seismic assessment 
on urban and territorial scale. The whole experimentation was conducted using only the potential of 
Grasshopper's VPL and possessing, as basic knowledge, the main concepts of machine learning and 
supervised learning, The VPL is therefore an effective tool to introduce and disseminate the topics 
and applications of artificial intelligence within the AEC sector, effectively decreasing the gap between 
domain experts and programmers. 


Keywords 
open data, VPL, Al, seismic assessment, CIM — city information modeling. 
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Introduction 
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within VPLs by reducing the knowledge required to apply them. These plug-ins enable the 
user to use Machine Learning and Deep Learning tools, enabling increasingly complex data 
processing practices. Applications range from design to optimisation in production process- 
es, Although in some applications there is no need for textual programming implementa- 
tions, VPL shows limitations in the long term. 


Urban Seismic Risk Assessment: the Italian Methods 


n relation to seismic vulnerability assessment at the urban scale, three majors schools of 
thought have been identified that aim to combine expeditious surveying with accuracy in 
assessment. These methods differ mainly in the type of data required and the accuracy of 
the analysis. In particular there is an inversely proportional relationship between analysis 
extension and accuracy assessment (the more accurate the assessment, the closer to the 
architectural scale). Statistical evaluations focus on the determination of vulnerability with 
reference to different main characteristics of the buildings in order to analyse their distribu- 
tion over the territory. Other analyses follow a mechanistic procedure where the structural 
behaviour is studied by simulating seismic actions on the building unit. As regards holistic 
analysis, it generally begins with an investigation of the urban growth of the fabric in which 
the building under analysis is located, then it recognises the construction components, maps 
the decays and analyses instabilities [Corradi et al. 2014; Calid et al. 2018]. 


Methods and Workflow 


Starting from existing studies on seismic vulnerability, the objective is to classify, through 
Artificial Intelligence mechanisms, the vulnerability of single building units using a few pa- 
rameters easily obtainable from qualitative visual surveys. This approach is based on the 
assumption that each building unit is a living organism with its own genetic code made by 
all its parameters. In literature we find similar approaches at the architectural scale of BIM 
models [Tono 2018]. Therefore, it is essential to use programming tools that allow sufficient 
data granularity for their treatment from the territorial to the architectural scale and the use 
of Artificial Intelligence tools. The use of VPL based on CAD environment allows to easily 
interrogate the geometries obtained from the initial data, as well as to visualize the results 
of the analyses through thematic three-dimensional maps. In the specific case of seismic 
vulnerability at urban scale, the use of VPL as modelling and analysis tool facilitates the AEC 
sector professionals in the design phase thanks to the available plug—ins. 
The research presents a workflow, developed with only the tools of visual programming (GH), 
to train a neural network using a dataset of seismic evaluations at the urban scale (statistical 
method). This approach belongs to supervised learning methods. In particular; a simple valida- 
tion will be performed by means of a linear regression with several variables that identifies the 
relationship between indices and vulnerability values thus allowing the prediction of vulnera- 
bility in other urban blocks. The workflow can be summarised as follows: 


E Downloading and importing the initial dataset; 

. Data processing according to the Simple Validation scheme; 
d Neural network training using the ‘Dodo’ plug-in; 

. Model validation (coefficient of determination R2); 

E Representation of the obtained predictions. 


Case Study: the Historical Center of Palermo 


To validate the proposed methodology, a dataset with the necessary characteristics was 
identified. The dataset comes from an open data work developed by the PalermoHub map- 
pers community [1]. In this work, a seismic vulnerability analysis based on statistical methods 
was made for more than 1500 building units in two areas of the historical centre of Palermo. 
The whole dataset was created exclusively on open data available online from different 
institutions such as ISTAT and the Municipality of Palermo [Vitrano 2017]. 
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Fig. |. 

a) Initial dataset (webgis) 
of Palermo historical 
center; 

b) VPL code to 

import geodata inside 
Grasshopper; 

c) Training dataset (in 
red) and test dataset (in 
green). 


The dataset is made of indices related to the period of construction, number of floors, con- 
struction material, state of preservation and the vulnerability of the building units. 

The file (available for online viewing) was downloaded as geojson and converted to a 
shapefile using QGIS. The conversion into a shapefile enabled the import into GH via the 
“att plug-in. A code was then developed with the aim to filter the indices and vulnerabili- 
ties of each individual building unit contained in the input shapefile. 
In the field of supervised learning, the method of the simple validation requires that the 
dataset is divided into two parts. The dataset destined to the training of the system usually 
constitutes 70-90%, the remaining part (30-10%) is destined to validate the training of the 
model of machine learning. In this case, the part assigned to training is 71.3% (DSI), the 
part assigned to validation (testing) is 28.7% (DS2). Within these two subsets, a filter was 
developed (via VPL) to separate the indices of building unit characteristics from the corre- 
sponding seismic vulnerabilities (fig. |). 

The open source Dodo plug-in [|] was used to train the neural network. Dodo allows to 
specify some significant parameters regarding the training process. In particular, the compo- 
nent ‘Supervised Training NN’ was used for the training of the neural network (it represents 
the phase of supervised learning). As input, the indices and the relative vulnerabilities of 
each building unit within the DS were given. The output was the neural network structure 
trained to identify the relationship between indices and vulnerabilities. Then ‘Run Neural 
Network' component was launched giving as input the trained neural network and DS2 
indices (test) to predict the vulnerability of DS2 building units. 


Results and Discussion 


The results of the predictions were used to create a representation of the buildings of the 
tested set DS2 (in blue) in comparison with the initial values (in grey) (fig. 1).Visual analy- 
sis of this 3D map suggested that the neural network identified the relationship between 
indices and vulnerability An analytical verification was then made using the coefficient of 
determination (both standard and reduced) which confirmed this result, returning a value 
of 0.99 out 1.00. By graphically analysing data distribution, the neural networks managed to 
fit the test data almost perfectly (fig. 2). 

However, the high values linked to the coefficient of determination are probably due to 
the statistical relationship between indices and vulnerability. The most significant limita- 
tions that have emerged from this experience are the ‘black boxes’ aspects of the com- 
ponent, and the absence of existing statistics that allow simple and effective comparison 
with other machine learning tools. Some potentialities emerged during this experimenta- 
tion, such as the ease in implementing simple experiences and the usefulness of VPL as a 
learning medium for the main concepts linked to artificial intelligence towards the use of 
more robust frameworks (as Tensorflow and PyTorch). It is therefore possible to consider 
GH as a digital carnet where a domain expert in the field of drawing and surveying can 
sketch a prototype Al model to identify the problem. Once this is done, the model can be 
implemented working with Al experts or by the domain expert himself after a learning 
and training phase. 
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Fig. 2. 

a) Training of the neural 
network with Dodo 
components; 

b) VPL code to display 
prediction results; 
Comparison of 
predicted (in blue) (c) 
vulnerability values and 


actual ones (in grey) (d). 


Supervised Training NN 


Conclusions 


There are still some open questions that constitute the next steps of the research. In partic- 
ular, how to link the information produced in urban surveys to the classification of building 
types, how to predict the internal distribution pattern of building units using spatial syntax 
analysis (since it is not possible in many cases to access the interior spaces). Furthermore, 
from an economic point of view it is possible to envision a model that can support the 
prediction of the cost of seismic retrofit and/or demolition interventions. 


Notes 
[I] http://palermohub.opendatasicilia.it/ (15th February 2021). 


[2] Dodo is a plug-in for Grasshopper developed by Lorenzo Greco. It is available on the online portal 'Food4Rhino' since 
23/11/2015. Last version: 23/02/2019. Link: https://www.food4rhino.com/app/dodo (| 5th February 2021). 
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V.A.I. Reality. A Holistic Approach for 
Industrial Heritage Enhancement 


Assunta Pelliccio 
Marco Saccucci 


Abstract 


The post-industrial heritage, characterized by heterogeneous, tangible and intangible factors, requires 
digital tools and a holistic approach to undertake the most appropriate enhancement process. Cur- 
rent virtual realities (Vr Ar Ir) allow the modeling of physical environments and the management and 
virtualization of a large and varied amount of data, thus helping to better understand the complex- 
ity of the real phenomenon. The paper proposes a method, named D.V.M.R. (acronym for Design, 
Virtualization, Modeling, Reproduction), which in four temporally consequential phases builds a tool 
capable of providing territorial, environmental, architectural and historical information of a case study. 
The method was applied for the design of a reuse of the brick factory, known as ex Sieci and located 
in Scauri in southern Lazio, owned by the Municipality of Minturno. The factory, which looks like a 
majestic cathedral on the sea, had in the past and still has a significant centrality in the social life of the 
local inhabitants. 


Keywords 
virtual realities, holistic approach, H-BIM, smart-service. 
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Introduction 


The European Union is increasingly convinced that cultural heritage can stimulate the 
sustainable development of nations in terms of liveability of the environment, social co- 
hesion, well-being, creativity and employment . This statement is even more significant 
in the case of post-industrial heritage, characterized by factories, chimneys and clusters 
of volumes necessary for industrial production, which has shaped the skyline of many 
cities and marked the culture of many communities. In the past, factories have attracted 
families and created a modus vivendi, made up of social and religious sharing, customs 
and traditions, generating their own genius loci. In the industrial landscape, the factory is 
the most important landmark, the visual and symbolic attraction of the recognizability of 
a community and therefore of belonging to the place. Today, however, the material and 
intangible cultural heritage of disused industries, consisting of both architecture but also 
religious and secular traditions, idioms, anecdotes still alive in the population, appears 
shaped by architectural skeletons that usually occupy vast urban and peri-urban areas. 
Sometimes these sites are highly polluting, negatively impacting the environmental sustain- 
ability indices (ESI). The current situation requires a very complex process of enhancing 
disused industries which must be based on urban redevelopment interventions capable 
of restoring both the tangible historical value, in terms of architecture, but above all the 
cultural, social and economic value of post industrialized communities. Today, digital tech- 
nologies are of great help to this end, due to their ability in modeling cultural heritage as 
a complex digital asset ensuring its understanding, consultation from different points of 
view and to the stakeholders. 

Furthermore, the combination of virtual reality (VR), augmented (AR) and immersive (IR) 
based on a holistic approach is the most suitable procedure for enhancing brownfields. In 
fact, on the one hand, virtual technologies help paradigmatically in the reading and inter 


pretation of heterogeneous territorial and historical data, the state of conservation and 
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usability of the material and immaterial value of the asset. On the other hand, the holistic 
approach allows to have a complete overview of the phenomenon. This paper applies 
the new graphic language, based on the use of the informative, interactive and immersive 
digital model, defined as an ‘smart model’, combined with a holistic approach in the acqui- 
sition and management of heterogeneous data with the aim of defining the best practice 
in the enhancement of abandoned industrial places. To this end, the brick factory, called ex 
Sieci and located on the seafront of Scauri (southern Latium), was chosen as a case study. 


D.V.M.R. Method: the Holistic Approach in VR-AR-IR Modelling of Cultural Heritage 


Since their origins, factories have developed their own inherently holistic graphic language. 
The design of the factories, in fact, required the representation of both the ‘architecture 
of space’, in terms of the permanence of man and his needs, and the ‘architectural space’, 
responsible for housing the machines and their functional systems. Today as yesterday, the 
reuse of abandoned industrial sites, which sometimes cover several hectares of urban sur 
face, need a new graphic language capable of visualizing, or rather virtualizing, the dimen- 
sionality and multiplicity of sites markers such as the territorial, historical and architectural 
factors. Nowadays virtual modeling, understood as VR, AR, IR, can dynamically and interac- 
tively manage large and heterogeneous datasets that are the informative, parametric and 
interactive structure of smart models. Such a complex data structure, ranging from spatial 
data to the smallest technological detail of the factory, must obviously also deal with the 
management of the relationship between each component and the functional sum of all the 
parts, thus following a holistic approach. In this perspective, the method proposed in the 
paper is able to virtualize the real object (referent) and the complex dataset associated with 
it, according to a holistic approach (fig. 1). The method has four temporally consequential 
steps, described below: 
|. The Design (D) of a geographical information model, GIS, which returns the environmen- 
tal and territorial context of the site under study as well as the consistency of the phenom- 
enon of abandoned industrial sites (polluting/non-polluting, recovered/abandoned, public/ 
private, etc.). 
2.The Virtualization (V) (signifier) of the real object (referent), obtained thanks to 3D digital 
surveys (laser scanning and digital photogrammetry). The point clouds exported thanks to 
geolocation in parametric software (H-BIM), become the semantic structures of the infor 
mative models. In fact numerical, material, technological and historical data are associated 
with each graphic object, according to the most appropriate ontological model. 

3.The Modelling (M) of intangible environmental phenomena (smart service). Thanks to IFC 
technology, the H-BIM model can be exported to other software capable of simulating for 
example the wind and solar exposure or acoustic analysis. 
4.The Reproduction (R) of the intangible cultural value of the site (signified). H-BIM is a fun- 
damental support for the immersive reality model because it is the box in which complete 
sensory experiences materialize through the reproduction of sounds, holograms, etc., thus 
helping people to be more aware of the cultural value of the site. IR also plays a key role in 
the decision—making process related to a post-industrial site rehabilitation project. 

The D.v.M.R. uses GIS and BIM as VR models as they produce realistic 3D models within 
which it is possible to virtually navigate thus aiding the analysis of the entire real phenome- 
non. Furthermore, in DVMR, the VR models (GIS and BIM) become physical prototypes to 
be used in software for simulation and calculation of immaterial environmental phenomena, 
which for this reason can be defined as Augmented Reality (AR). This type of simulation is 
particularly important for analyzing and evaluating degradation phenomena on the facades 
of historic buildings and the design of functionality, through new and more suitable desti- 
nations in use. Similarly, in DVMR the VR models (H-BIM), are the box within which the 
immersive reality (IR), which ranges from entertainment, education, promotion and acces- 
sibility for people with disabilities, helps the perception of the intangible value of the asset. 
The D.VM.R described above was applied in the enhancement process of an important 
abandoned industrial site in southern Lazio. 
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Case Study: Abandoned Brick Factory in Scauri 


The brick factory, called ex Seci, located in Scauri in southern Lazio, was born on a small 
pre-existing industrial settlement, as a branch of the Albizi furnace in Remole di Pontessieve 
(Florence) located on the Sieci stream from which it takes its name. The factory is part of 
a very delicate landscape both for its natural features and the numerous archaeological and 
historical evidence present in the area. The main elements are the coast called ‘Riviera di 
Ulisse’, the archaeological area of the ancient Roman town called Minturnae, the Via Appia 
and the mouth of the river Liri (Garigliano) on the border between Lazio and Campania 
Felix. The plant covered an area of 5 hectares with about 50,000 cubic meters of buildings 
and was connected to the sea by a destroyed wooden walkway, which facilitated the loading 
and unloading of products, marketed by sea. The main body of the factory, one of the few 
remaining, is spread over three floors above ground and rests on a base defined by a series 
of round arches. The building has a clear neoclassical style characterized by empty/solid 
rhythmic sequences, further marked by a series of pilasters which define regular modules 
and shape all the facades. Each floor has, between the piasters, three openings (windows) 
separated from each other by masonry completely covered in brick. On the ground floor 
there are two Hoffman ovens, still well preserved, which almost entirely occupy the longi- 
tudinal layout of the factory. In the external space, some further smaller volumes complete 
the area together with the chimney still in fair condition today. The construction system is 
in load-bearing masonry and the building is entirely made of bricks to enhance the product 
for which the factory was born. Over the centuries, due to the bombings of the Second 
World War and the reduction in production, many volumes have been destroyed. Current 
in the eyes of ordinary people, the factory looks like a ‘cathedral on the sea’. The centralit 
of the factory in the social life of Scauri is preserved by the locals who still today occupy 
the square in front with a weekly market, thus guaranteeing, albeit weak, a form of life’. The 
factory is currently owned by the Municipality of Minturno. 
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Application. Scan to BIM, Smart Cultural Heritage Services and Visitor Experience 


The reuse project of the brick factory has required the application of the DVMR. 

In the first phase, the Design (D) of the geographic information system through QGis was 
fundamental to analyze and understand the territorial context of the site.The GIS, in accor- 
dance with the ontological structure of the ISO/TC21 | standards, collected data on three 
different territorial levels: a) the landscape, with the indication of the vulnerability and con- 
sistency of the industrial sites already existing in the territory (in terms of pollution, dimen- 
sion, renovation, disuse, etc.); b) urban planning, with particular attention to the regulatory 
but also socio-economic aspects; c) area of the case study, getting of general, dimensional, 
technological, historical data, etc. 
Scan to BIM:Virtual Reality (VR) for the definition of a geometric/parametric model: The Vi 
tualization (V), in the second phase, performs the procedure known as SCAN to BIM.Th 
size and internal spatial distribution of the plant suggested the use of a digital photogram- 
metric survey from a drone (returned with the DroneBase X1000 Mapper con fotocamera 
Sony Alpha ILCE -6000 | 6mmf/7.1,by double swipe flight plan at a height of 40 m from the 
ground) to reproduce the external volume and a laser scan survey (returned with Leika 
BLK 360 by the scans alignment of a specially designed grid) for the interior Thanks to the 
aser scanner survey, the superfetations, the structural consistency and the state of decay of 
the finishes and structural parts were analyzed. Furthermore, through georeferencing, point 
clouds can be imported into an H-BIM system and the virtual reality model has been built 
according to the ontological structure of the UNI 1337-2017 standards (LOD, LOG and 
LOIN). The virtual model of the factory ‘Ex Sieci’ is characterized by Lod 400/AS BUILT. 


+ 


Smart heritage services for environmental analysis: Modeling (M) of natural phenomena in vir 
tual environments, in the third phase, is an example of AR technologies. The HBIM parametric 
model exported, thanks to IFC technology, in a computational fluid dynamics software allows a 


multi-criteria analysis capable of evaluating the effects of wind on the facades of buildings. The 


o P 
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model can be imported, in fact, into a virtual wind tunnel, where the average wind direction 
and intensity are taken from the Annals, and the wind impact on the facades can be virtualized 
with sized voxels. In addition, taking advantage by geolocation, H-BIM performs energy anal- 
yses and shows the impact of solar radiation on the facades of buildings during the day. The 
visualization with false-colour of sunlight on the building walls highlights the surfaces with the 
greatest exposure to the sun, providing a significant indication in energy efficiency measures. 
Similarly, it is possible to virtualize in H-BIM the execution of different acoustic analysis scenar- 
ios, which are of great help in a re-functionalization process. Environmental analysis through 
Augmented Reality is a very helpful smart heritage service. 

Immersive reality (IR) for the intangible cultural heritage: The last phase involves the use of 
H-BIM in the enhancing immaterial value of the place. The design of the ri-function of Hoff- 
mann ovens, very complicated to manage, has combined the IR in two different approaches. 
One gate of the oven, with interactive screens, holograms and sounds, which recall the sea- 
scape in which the factory is located, aims to attract the interest of children to the natural 
landscape that surrounds the factory; the second gate, narrating the historical events that 
characterized the factory and the surrounding area during the Second World War, intended 


to attract the interest of adults. 


Conclusion 


The post-industrial heritage, which has marked the cultural and social life of many commu- 
nities, today requires an impressive process of enhancement that must analyze all the differ- 
ent factors that come into play with a holistic approach. To this end, it is important to define 
a method that, on the basis of the potential of new technologies, is able to virtualise both 
the material and immaterial aspects. The method proposed with the abbreviation DVMR 
was conceived for this purpose and tested on a real case study, the brick factory located 
in Scauri. This first application, which required the acquisition and processing of many data, 
both territorial and historical—artistic, will be implemented with additional technologies, 
3D viewers, with the aim of improving the sensitive perception in the enhancement of the 
intangible cultural value of the site. 
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Abstract 


The year 2020 should have been one of confirmation of the continuous growth in the “cruise tour- 
ism sector”: new ships for delivery, new constructions, and the design trend of the gigantism of these 
means of transport. If, at first, ensuring a correct perception of orientation on board these new ships 
were a complex challenge and not yet fully resolved, today with the outcomes of the Covid pandem- 
ic it appears even more and more complex. 

The logistical communication of the past now seems to be no longer sufficient: the collapse of cruise 
bookings has introduced serious problems in the sector which must not only rethink new models of 
internal organization on board, but also reconsider how to regain the trust of potential passengers, 
frightened at the idea of getting on a ship. 

This research introduces some reflections and proposals shared with industry operators and aimed 
at the reformulation of approaches and languages in order to bring potential customers closer to 
the Cruise Experience. 


Keywords 
visual communication, multimedia approaches, complex structures, cruise transport. 
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May 2020: harbour cities change their perceptive profiles, transforming the usual view of the 
harbours of large cruise ships into a visual void, replaced by the silhouettes of the giants of 
navigation at anchor outside the port, as if they were ghosts in memory of a remote era. 
Venice is experiencing a return to the past, the waters of the lagoon become clear again and 
the perspective of the Grand Canal is finally free from the closures caused by the transits 
of “oversized” ships, to which this wonderful town resigned itself, 

The pandemic makes evident the disproportions and overruns of the limit, the sizes of the 
cruise giants now appear in all their excesses, as does the crowd pouring into the collective 
spaces, that cause a real inevitable gathering; some cases of infections on board accelerate 
a process of disruption of the system, visualized and amplified by the media, which transmit 
images of health emergencies. 

September 2020: Summer gives the illusion of an end to the pandemic and a conse- 
quent possible recovery of the cruise sector, which timidly reappears with advertising 
campaigns all aimed at reassuring potential users, through alignment with the restrictive 
anti-Covid measures. 

Even if it has been a false hope and the resumption of the spread of the virus has blocked 
at the beginning the revival of the economic induced by sea travel, however the story has 
highlighted the substantial obsolescence of the communication systems that on board wel- 
come and support passengers: panels and paper brochures, voice information and electron- 
ic touch screen devices, that is the combination of screen and digitizer, in order to allow the 
user to interact through the simple and immediate use of their fingers. 
f for the analog component the potential and limits relating to application areas and refer 
ence targets are now clear and consolidated, the reasoning for touch screen instruments, 
until recently considered a definitely positive model of effectiveness, efficiency and balance 
of the cost-benefits ratio is different. 
What intervened, on the contrary, to undermine this certainty? Simply the awareness of 
the health risk that occurs whenever several people touch the same surfaces with their 
bare hands: Sars-Cov-2 virus, in fact, can be also transmitted through the contamination of 
unwashed and / or disinfected hands. 
Here, therefore, that the search for alternative forms of communication develops from a 
hygienic-sanitary emergency situation and the occasion of the crisis in the cruise sector 
represents an opportunity to experiment and mutate the techniques already in use in other 
cultural and geographical areas; not only but, it also represents an opportunity to update the 
contents, starting from the concept of customer—caring. 
t could be said that the new frontier is given by the awareness that It is not a problem of 
solving a specific question that is different from time to time and not systematizable, but of 
undertaking a path to assure and protect the well-being understood in a global sense of 
those who use the cruise travel system; this proactive attitude translates into the formu- 
ation of a methodology of general value, which aims to draw up guidelines in compliance 
with certain assumptions, which are essentially the following ones: 

e To convey a sense of attention to the passenger at 360°; 

e Not to limit to signal attention to the Covid emergency; 

e To promote the full knowledge of the medium; 

e To promote a full knowledge of all its functional values; 

e To incorporate hygiene behaviours into good practices on board; 

e To avoid conflict with mandatory signs. 

Above all, of course, it is mandatory to make the promoter recognizable, or to personalize 
the communication system, from time to time adapting it to the identifying characters of the 
specific shipping company. (MLF) 
The design approach will necessarily have to combine — as mentioned — analog media with 
multimedia and assisted technologies; in particular, QR codes will be used, now consolidat- 
ed in their wide-range use, and on-board communication will be then integrated through 
technologies based on Projected Augmented Reality, on an infrared light system detected 
through the integrated combination between hardware / software, as well as on holograph- 
ic representation (fig. |). 
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Fig. |. Technology based 
on projected Augmented 
Reality that allows you 
to merge the real world 
with the virtual world. 
Each surface in the 
detected environment 
can become a possible 
screen. 


These choices are motivated by the search to obtain a form of communication that might 
reach users in a way immediate and effective as possible, in order to always respect tar 
gets and purposes in a contextualized manner It is essential that the result of the design 
choices is a system of documents of rapid perception and of univocal, simple and memo- 
rable understanding; alongside these requests, as for what concerns the present moment, 
a further request to be satisfied is given by the respect of a careful hygiene, where, con- 
sequently, no physical contact is foreseen, not even only tactile. 

These are certainly alternative and above all updated modes, which offer a valid alter 
native to all systems based on the touch screen, which appear immediately obsolete 
and ineffective. 

Overall, the Projected Augmented Reality represents a fusion between the real world and 
the virtual one with its effective and captivating perceptual yield, which is implemented 
thanks to a technology with latest generation projectors and the integrated combination 
of hardware and software, with the visible structured light, the “intelligent” scanning and the 
privileging of the point of view of the projector; the scanning of the visible structured light 
allows the alignment and correspondence “pixel by pixel” among the elements present in 
the real environment and the captured digital image, as if the latter had been taken with the 
optical parameters and the point of view of the projector 

The high-resolution projection of the image — automatically aligned with the objects 
present in the real world — thus assumes a significant importance, as it allows to transmit 
the information processed and converted into 3D maps directly on the real surfaces 
that become possible screens. Dedicated softwares therefore make it possible to apply 
procedural effects and projector control to produce captivating and engaging visual ex- 
periences through the creative and synchronized projection of visual and audio effects. 
Games of lights, colors and shapes automatically adapt to real surfaces, thus enhancing 
their scenic depth. Ultimately, this form of augmented reality uses the projected light to 
“increase” the contents to the reality without the aid of viewers or devices necessary for 
their vision, as is the case, on the contrary, in other forms of AR. This peculiarity allows 
its use in multiple areas of application: from the individual artistic installations to the 
promotion of the territory, from the urban redevelopment of “non—places” to being an 
integral part of wayfinding for orientation in real-life spaces as in public places and on 
cruise ships (fig. 2). 

Furthermore, some contemporary communication systems of wide diffusion and use offer 
the starting point for a further series of application possibilities; above all, the use of artificial 
intelligence, with characteristics particularly suitable in certain situations and for targeted 
target audiences [1]. 

“Where did | park the car?” “What is the shortest way to reach the goal?” “Which is the 
most scenic route?” 


fas] 
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Fig. 2. Integrated on- 
board communication 
through the use of 
technologies based on 
Projected Augmented 
Reality. 


In everyday life, the contribution of tools that interact with us on the basis of the principle 
of artificial intelligence as the realization of the human ambition to be able to create a 
relationship between man and machine, between automation and rational thinking, is not 
only consolidated, but even almost obvious. Computers and ‘smart’ devices provide us 
with virtual secretaries that simplify and support daily activities. (RT) 

Naturally, it is a matter of distinguishing which type of automation can be usable and con- 
sistent with the communication and information needs necessary on board cruise ships; 
as Is known, automation Is structurally divided into two fundamental theories: weak Al and 
strong Al, defined by the scholar John Searle this definition allows to precisely parametrize 
and circumscribe the areas of action. 
Very briefly, the differentiation between the two types of Al is based on the different 
action that ‘imitates’ the mechanisms of the mind, assimilated to computer programming: 
the stimuli that reach the brain lead to an immediate reflection of purposeful thinking (a 
‘reasoning’), from which a consequent and coherent action derives. The machine, there- 
fore, has the purpose of imitating and simulating the reasoning produced by the human 
mind, at different levels, or ‘limiting’ the activity to the possibility of answering questions 
that derive from the careful and timely analysis of the data held by the tool, or trying to 
become itself a sort of autonomous intelligence, which not only learns from experience, 
but interacts with other non-quantifiable factors, such as emotions or the subjective way 
of expressing oneself. 
t is quite immediate to understand how the type of interest here is the first, since 
the technological system, necessary for communication and information intended for 
cruise passenger, works on specific and repeatable questions, in a completely similar 
way to what happens with the ‘virtual secretaries’ which we are already used to: in fact, 
in the case in question, the applications requested from the machine, are limited to 
understanding and solving specific problems through answers directly derived from the 
information in possession. 
From all the considerations that emerged, it is clear the need to bring together all the 
different categories of media, from the traditional analogue ones, to the more contem- 
porary visual ones, up to the interaction with Al; the range of possibilities should make it 
possible to reach the widest target of users, regardless of the level of computer literacy, 
culture, age, possible disabilities and so on. 

There are many possible examples to draw from: the attached images bring only a very 
small part of already tested implementations (albeit in other contexts), which could be 
revisited and introduced as a customer-caring system on board cruise ships. Lastly, it is 
recalled that the ideas and arguments contained in the text refer to the research in prog- 
ress thanks to the partnership of Grandi Navi Veloci, to the collaboration with the Centro 
del Mare of the University of Genoa, with the PhD in Science and Technology for the Sea, 
as well as with some companies specialized in the research and implementation of the 
technologies described. 
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Specifically, the collaboration with Grandi Navi Veloci led to the awareness of a necessary 
segmentation of the communication offered on board; if, in fact, the age group of the most 
widespread cruise target is undoubtedly literate and active from the use of devices and 
IT tools point of view, on the other hand it is also true that a communication based on an 
almost dialogic use of the “machine” could even cover also the less skilled users especially 
with multimedia. This, therefore, is the new challenge: the use of Al at levels of immediate 
understanding of information, to support each passenger with a virtual assistant in a tar 
geted manner [2] (ER). 


Notes 


[I] In particular, a part of the study presented here is part of the project entitled: welcome on board: integrated habitat 
communication in cruise ships, funded by University Research Funds — University of Genoa 2019, responsible R. Torti. 


[2] The research presented is the result of the joint work of the three authors; the individual contributions can be traced back 
to the authors by the presence of the initials at the bottom. 
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Abstract 


This research work into Evolutionary Computing field aims at improving a dataset of algorithmic gen- 
erative definitions able to return an optimized ‘semi-ideal’ curve that best fits a generic reality-based 
profile, starting from some of its points. This paper shows GAs_Genetic Algorithms applications 
especially with regards to study, interpretation and definition of generic polycentric curves. Current 
VPL tools (Galapagos-Rhino, McNeel) allow to test Evolutionary Theories for problem solving and 
decision making in architectural research field. According to a human driven approach, an operator 
defines GENOME, FUNCTION and FITNESS to drive the Evolutionary Solver towards optimized 
solutions. Some case studies from Historical/Existent Architectural Heritage are used to show how 
GAs can simplify the digitalization process and big data interpretation. 


Keywords 
genetic algorithms, evolutionary computing, geometry, PolyArc, cultural heritage. 


J TTT GENE POOL INPUT Q) SOLUTION ® 
Dnt @ gee 
Swe 13 RITA 
reality-based VPL GAs FA Se O q A X% rules comparison 
models Pye s — 
m FITNESS — 


> 
Evolutionary Computing GENERATIONS interpretation 
PI 


OPEN GENERIC _ P=? 
polycentric curve < 


COMPARISON=DISTANCES 
reality-based/semi-ideal profiles 
cone 7 N 


P=n 
CLOSED GENERIC “¿=p 
polycentric curve C=n 


403 doi.org/ | 0.3280/0a-686.64 


Introduction 


This research work into EC_ 


Evolutionary Computing field aims at developing a dataset of 


algorithmic generative definitions (VPL), to return optimized ‘semi-ideal’ curves that best fit 
reality-based profiles (generic polycentric curves from point cloud segments). This paper 
especially shows testing of GAs [Holland 1992] for vaulted systems study. EC is a subfield of 


Al aimed at iterative, continu 


In architectural and engineering fields, it is possible to distinguish applications about op 
mization of architectural-urban design [Buffi et al. 2020; Canestrino et al. 2020, Palma et 
al. 2020], analytical-structural applications [Grillanda et al. 2017, Khan 2015], manufacturing 
complex elements-systems [Zaremba 2016; Coutinho 2010; Limonge et al. 2010], analyti- 
cal-geometric applications to rebuild and compare shapes [Bianconi et al. 2018] and, more 
specifically, about ovals interpretation [Santagati et al. 2018]. 
Moreover, similar approaches are managed with different tools to optimize the curves 


and surfaces interpretation 


zara et al. 2019]. 


ous and combinatorial search for optimized solutions. 


+ 


LO 


that describe historical architectural elements, according 


stylistic features of cultural heritages and geographical contexts [Samper et al. 2020; Lan- 


These approaches are also potentially aimed at supporting Al processes [Sim 2020]. This 
VPL algorithmic definition allows to construct all kind of profile starting from reality— 


based elements: the only input parameter is the points number It is the main advan 


of the process. 


Methodology and Tools 


models and related informat 


Digitizalization of existent architectural elements is a process to provide a system of 3D 


+ 


tage 


ion (parameters/geometric variables, construction techniques, 


materials). About Historic Heritage, these properties can be extrapolated through a direct 


analysis of architectural elem 
rules. The parameterization 

used to generate the revolut 
201 9a]. Digital translation of 


adaptable configurations [Ca 


+ + 


architec 


tural element, providi 


ents (survey) or from specific treatises and historical manuals 
of pointed arches and polycentric curves (vertical sections), 


ion pointed domes, is one of tested approaches [Capone et al. 


geometric-mathematical rules simplifies the parameterization 


geometric genesis) of complex architectural systems and allows to model variable and 


pone et al. 2019b]. Another approach is based on VPL models 


built on reality-based profiles used as “input parameters”, to compose the wireframe of the 


ng a 'semi-ideal' model closer to the real element. Then, differ 


ent approaches and tools (VPL/C++) were compared to identify ideal curves and surfaces 
that best fit point clouds segments [Lanzara et al. 2019]. 


We have applied these approaches to define generic polycentric curves, open or closed, 
from reality-based profiles using GAs. This contribute shows applications about closed sym- 


+ 


metrical polycentric curves 


and open polycentric curves that could be domes profiles. 


Ovoidal domes can be generate as revolution surfaces or they can be shaped from curves 


network. The axes dimension 


+ 


In fact, it is not uniquely possi 


s are not sufficient as input parameters to model these shapes. 
ble to identify and to draw the specific oval that best describes 


a reality-based profile according to the lengths of its axes only. Starting from the same pair 
of axes, it is possible to generate infinite ovals and one ellipse [Dotto 2002, p. 14]. 
Current VPL generative-algorithmic tools (Galapagos-Gh component, Octopus-Gh plug- 


+ 


tionary Solver 


+ 


towards 


component aimed at built a 
reality-based profile to extr 


+ 


type of ideal profile best fits 


n, Rhino, McNeel) allow to test Evolutionary Theory (Darwin 1859) to support problem 
solving and decision making processes. According to a human driven approach, the param- 
identification to define GENOME, FUNCTION and FITNESS allows to drive the 


an optimized solution. 


GAs calculates the optimal position of the end-points (GENE POOL) of a PolyArc (VPL 


sequence of tangential continuous circular segments) along a 


apolate a ‘semi-ideal’ polycentric curve. A direct comparison 


between the ‘semi-ideal’ curves (GA) and the ideal curves (rules) allows to establish which 


and decodes the reference curve. The number of points can 
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Fig, |. GA to interpret 
and define CLOSED 
POLYARC — OVAL 
profiles (impost and 
intermediate sections) 
of the Church of S. 
Giovanni Maggiore's 


dome. On the right-top: 


comparison between 
'semi-ideal' profile and 
rules; on the right— 
bottom: comparison 
between reality-based 
profiles, oval and ellipse, 
(authors elaborations). 


be random or deriving from a critical interpretation of the reference subject. Symmetrical 
distributions (entrances, chapels, mosaics, niches or structural-decorative elements) sim- 
plifies the decomposition of a PolyArc into its segments and allows to define hypotheses 
about the specific oval profile. If the curve is closed, the points number (GENE POOL) is 
the same of the arcs number; if it is open, the number of points is the same of the number 
of arches/centers +1. 

The Evolutionary Solver combines points by minimizing the distance (Mass Addition)be- 
tween reality-based curve and PolyArc (FITNESS) to optimize their overlap. FITNESS de- 
fined and tested to select the optimal solutions, allow to minimize the sum of the distances 
between the ideal curve and the reference one and the average distance between curves 
and they allow to maximize the identification of points whose distance from the reference 
curve is smaller or equal to a given limit value.A chromatic gradient distinguishes the points 
along reality—based profile according to their distance from defined ‘semi—ideal’ curve and/ 
or the ideal configuration that best fits it: for a value of 0, points are green; for higher values, 
points are red. 
Once the ideal circular segments of the PolyArc have been identified, it is automatically 
possible to extract the whole circumferences and the position of their centers for each 
circular segment. Finally, it is also possible to compare the defined ‘semi—ideal’ PolyArc and 
its distributive layout of the centers with ideal profiles (curves built starting from centers 
along diameters). 

The main advantage of this definition is to use the points number as the only input param- 
eter: the difference between closed and open ‘semi—ideal’ polycentric curves is to commu- 
nicate this condition by simply using a Boolean Toggle (True/False). Therefore, a single defi- 
nition allows to analyze, interpret and define a generic open or closed polycentric profile. 
Another important advantage is to use the same GENE POOL (points along reality-based 
curve) to generate different curves. The main advance, in itinere, of this research activity is 
also about study and definition of analytic curves, e.g. conics, only starting from points along 
reality-based profile. 


py 
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Fig. 2. GA to interpret and 
return OPEN POLYARC 

— POLYCENTRIC ARCH 
(vertical section/meridian) 
of the Church of S. 
Caterina a Formiello's 
dome. On the right-top: 
comparison between 
‘semi—ideal profile 

and rules; on the right- 
bottom: comparison 
between reality-based 
profiles, oval and ellipse. 
(authors elaborations). 


Applications 


GA for OVAL interpretation has been tested to verify the impost and the intermediate 
sections of the ovoidal coffered dome of the Basilica of S. Giovanni Maggiore in Naples. 

A double symmetrical oval (four centers) is clearly recognizable with the naked eye by ob- 
serving the extrusion profile of the molded frame crowning the drum and corresponding 
to the impost curve of the coffered intrados of the dome. However, the intermediate oval 
profiles are similar to ellipses. 

Although ellipse and oval are conceptually and analytically different curves, for centuries 
they generate a “conflict” (Migliari 1995): the main reason lies in their formal similarity, 
often causing interpretative misunderstandings about the attribution of these shapes to 
geometric elements. 

Figure | shows comparisons between ‘semi-ideal’ ovals, Serlio's rules (1584) and the elliptic 
profile that best fits the intermediate sections. Unlike the impost profile, the algorithm cal- 
culates a minimal difference between intermediate sections and ellipse. However, also the 
presence of lacunars would confirm the oval shape for the intermediate sections: in fact, 
oval allows regular offset. 

About modeling of hemispherical pointed domes characterized by circular section, we have 
translated in VPL the geometric rules provided by Serlio and by Palladio; for polycentric 
vertical sections (curves composed by a series of continuous arches in tangency and cur 
vature), the rules provided by Fontana and by Scamozzi; for pointed arches (different ratios 
between ‘arrow’ and radius), the rule illustrated by Vittone [Capone et al. 201 9a]. Therefore, 
this comparative approach was tested on a series of revolution domes of Historic Architec- 
tural Heritage in Naples. 

In particular GA to interpret and build POLICENTRIC ARCHES was tested on the pointed 
dome of the Church of S. Caterina a Formiello to define the layouts of centers subtending 
one of its vertical section (without ‘vertex’). 

By comparing Serlio, Fontana, Vittone and Scamozzi's rules and the reality-based sections, 
we can state that the reality-based profile does not match any of the constructions of 
historical treatises actually translated into VPL, neither the ellipse. However, the difference 
between reality-based profiles and ideal curves is not minimal: therefore, it does not de- 
pend only on structural problems or constructive irregularities, but also on different stylistic 
approaches (fig. 2). 


intermedied section Polycentric OPEN curve 
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Conclusions and Future Works 


The main progress (in itinere) of this study is to test GAs to interpret and define analytical 
curves (e.g. conics) and generic profiles characterizing also modern and contemporary ar 
chitecture. In addition, other future advances can be about deepening the accuracy of the 
models, improving current VPL definitions and testing other types of algorithms [Gatti 2020]. 
The semi-automatic interpretation of complex elements simplifies their parameterization ac- 
cording to interoperable logics (VPL/BIM—HBIM). Furthermore, this approach is also aimed to 
test geometric-speculative approaches and to inspire studies with different goals. 
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REPRESENTATION CHALLENGES 
Augmented Reality and Artificial Intelligence in 
Cultural Heritage and Innovative Design Domain 


The Drawn Space for Inclusion and 
Communicating Space 


Anna Lisa Pecora 


Abstract 


In the last twenty-five years, some studies have been analyzing the virtual reality potential for special 
education. They show the virtual environments as a valid communicative medium and a safe space 
where ASD people can experience new situations without limits of ‘in vivo’ experiences. Often, prob- 
lems with space can complicate many aspects of everyday life. Referring to Hermelin and O'Connor's 
studies, the difficulties involving the autistic clinical frame are connected especially to perception 
deficiency, therefore the VR can become valid support, for people with ASD, improving relationships 
with space, with ourselves and with others. My study tries to provide a guideline tool for a human— 
centered VR design. 


Keywords 
virtual reality (VR), inclusion, autism (ASD), drawing, representation. 
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Introduction 


n recent years, the drawing disciplines, supported by the widespread of new technologies 
progress, play a fundamental role in the interdisciplinary mediation of the communicative 
field. Up to now, studies about special education and VR have been led by medical and 
psychological disciplines, disregarding perceptual aspects about the relationship between 
the user and the virtual space. Furthermore, a wrong interpretation of spatial configura- 
tion can provide various types of discomfort in both, typical and non-neurotypical users. 
fVR can mediate the relation with the real world, its visual codes must be managed with 
expertise, in order to provide a customized visual language on specific user needs. There- 
fore, the ICAR 17 disciplines can embody a valid guide to drive the perceptual process 
in virtual reality. 


VR Spaces for Learning and Inclusion 


In the last twenty-five years, the rapid development of virtual reality technology has 
improved the simulation quality, opening new applicative frontiers. Their communicative 
power lies in the potential to provide a perceptual realistic experience where It’s possible 
to activate the same cognitive dynamics available in the real world. Therefore, in special 
education, the immersive experience should support the specific characteristics of the 
autistic perceptual system and its atypical responses to sensorial stimuli. Some impair 
ments can create visual distortion complicating the reading of spatial clues; for example, 
they can inhibit the understanding of depth, dimensions, shapes and relative positions 
[Bogdashina 2015, p. 29]. Other problems can occur in recognizing the space limits or in 
the relation between the viewer's body and other objects. This problem can arise during 
a VR experience due to the specific characteristics of virtual navigation. Virtual reality 
uses panoramic images, also called 360 degrees images, as: photorealisti 


tic renders, photos 
or videos; therefore, since the frame limits exceed the visual field limits, the experience 
involves not only the vision but also the body, even in the simple head rotation. This way, 
the user turns from a static sight to a dynamic perspective [Rossi et al. 2019]. Usually, 
these relations between vestibular and visual apparatus help the wayfinding and the un- 
derstanding of the space, but in autism, they can provide confusion and stress due to the 
disorder of the “afferent couple” phase [Russell 1994]. For this reason, the majority of the 
research about VR and autism, attempt to create a safe space where experience new sit- 
uations without the limits of the physical world. Cultural heritage, for example, museums 
or archeological sites, often are uncomfortable spaces for autistic people, because they 
can provide overstimulation of senses, causing a painful and stressful experience for peo- 
ple with hypersensitivity. Using the potential of virtual reality, it’s possible to communicate 
cultural information customizing the input data on the specific user needs helping, this 
way, the dialogue between the space and the viewer Moreover, since the virtual expe- 
rience involves the user in the first person like in the real world, the emotional aspects 
play a strategic role, enhancing the learning process with a playful incentive. In fact, most 
of the learning experience for special education use applications from the gaming world, 
without customizing perceptual inputs on autism needs. Therefore, the research lacks in 
details concerning useful aspects about the perceptive response to the stimulus coming 
from the designed virtual space. Most of the observed studies are visually overloaded and 
unfitted to the cognitive needs of ASD people. The majority of research, in fact, points 
out psychological and technical aspects without considering visual factors involved in the 
interaction between autistic people andVR Trough the comparison between the scientific 
researches published about VR and autism, | identified four taxonomic categories in order 
to analyze the state of the art about this topic: 

Usability tests: valuating ASD users’ answers. 

Life skills: learning of basic competencies useful in daily life. 

Social skills: understanding of social situations. 

Special education: VR application in schools environment, mainly for language learning. 
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Fig. | . Cobb's experiment 
(1995). Frame of the 
virtual ski slope from 

the user's point of view 
and scheme with the 
main depth clues and 
target elements. Graphic 
drafting by Anna Lisa 
Pecora. 


Fig. 2 . Strickland’s 
experiment (1997). 
Street scene and scheme 
with the main depth 
clues and target elements. 
Graphic drafting by Anna 
Lisa Pecora. 


Visual Clues Improving Space Perception 


There is a huge difference between watching a real space and watching a drawn space. In 
the first case, space directly communicates its morphology without interferences, while, 
watching an image means to see a subjective interpretation of reality. Also, the virtual en- 
vironment develops by a drawing action, therefore the perceived image is double filtered: 
first by the designer, who translates his personal view of the world in graphical signs, then 
by our brain, that, according to the personal experience, decode the figurative codes in a 
mental image. Thus, the final result depends on multiple factors, some depending on the 
person, others on the picture characteristics. Usually, we distinguish three main categories 
collaborating to the space interpretation and, in detail, to the depth: pictorial depth cues [1], 
physiological depth cues, motion depth cues. Jerald Jason adds another category: contextual 
distance factors, related to the environment and psychological influence on our perceptual 
behavior [Jason 2016, pp. 122-123]. He explains that: “perception and action are closely 
intertwined. Action-intended distance cues are psychological factors of future actions that 
influence distance perception” [Jason 2016, p. 122]. This aspect is particularly relevant if re- 


ferred to autism where the answer to sensorial stimuli can be altered by an atypical behavior 
of the perceptual system. Moreover, this impairment can cause states of anxiety and fear, 
interfering in the right reading of depth cues. Consequently, designing a virtual space, it’s im- 
portant to take into account psychological aspects influencing physiological and projective 
factors when managing graphical signs. For example, the environment should be character 
ized by simplicity and clear spatial clues. Referring to this aspect, an example is one of the 
first experiments led by Sue Cobb and the VIRART team. There are no descriptions of the 
virtual environment, but, watching the published photos, it is possible to deduce some visual 
aspects interfering in the perceptual process. The VE represents a ski slope characterized by 
a simple framework where objects have basic geometries easily recognizable (fig. 1). The 
highly prevalent white produces a flat configuration where it's difficult to understand the 
perspective. For this reason, the use of some graphic depth cues is crucial: a curb on the 
right and a schematic ski lift on the left side, provide two linear clues that, converging toward 
the frame center, become important depth gradients. In fact, the angle of graphic signs on 
the vertical and horizontal axis is a key element for tridimensional perception [Arnheim 
1997]. The distances and dimension reduction of ski-lift elements offer other depth gradi- 


all 


Fig.3. Tables. Guidelines 
for an autism—friendly VR 
design: perceptual aspects 
and cognitive aspects. 
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ents in the observed image; acting in the same direction, they enhance each other because 
“the more the gradient is regular, the stronger their effect acts” [Arnheim 1997, p. 227]. 
We can find another example of communicating virtual space in Strickland’s experiment, 
published in 1997; that is one of the first immersive experience for autism lead with an 
RV-HMD. It could be taken as a reference for its way to put in evidence target elements 
and depth clues. Strickland, describing the environment design, explains that: “The virtual 
world was a simplified street scene consisting of a sidewalk and textured building shapes. All 
motion objects such as people, animals, and objects in the sky were removed. Periodically 
one car, whose speed could be changed, would pass the child standing on a sidewalk. The 
contrast was kept low in the scenes with gray being the dominant color The low quality of 
the headset screens provided a less detailed environment automatically. The cars, the focal 
point of the test, were presented in bright, contrasting colors [...] red and blue” [Strickland 


+ 


Dorothy 1997, p .4]. Only later, another visual stimulus is introduced: a stop sign is moved 
to different parts of the tracking area during the later tests and the children are asked to 
find it and stopping there (fig. 2). In this kind of configuration, the environment works like a 
neutral background helping to focus on visual targets. Otherwise, the oversimplification of 
morphological spatial signs creates difficulties in evaluating distances. It's possible to improve 
the information about the spatial reference using some graphic solutions without exceeding 
in perceptual overload; for example, acting on the textures or using some objects like land- 
marks in the scene. Sue Cobb's experiment, in 2002, uses a grid pattern for the virtual cafè 
floor, that works like a spatial coordinates reference. Moreover, they found that watching 
part of the avatars body, enhances the sense of embodiment and supports the under- 
standing of the avatars size during motion [Cobb 2002, p. | 7]. The sense of embodiment, is 
related to “presence”, also defined as “the sense of being there”, “inside a space, even when 
physically located in a different location” [Jason 201 6, p. 46]. Because ASD people frequently 
show proprioception impairments, they have difficulty feeling their body physically acting in 
the virtual environment as if they were performing the task in the real world. Giving the hu- 
man height to the point of view, provides a familiar appearance to the framework improving 
the sense of presence and the willingness toward the virtual experience. For the sake of 
narration, | could only describe here a brief example of the guideline tool | compiled for 
designing an autism—friendly VR. The attached tables (fig. 3) show part of the developed tool. 
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Conclusions 


The physical space holds the complex theoretical and morphological relations between 
different elements as colors, lights, textures and patterns. So, only through the right graphic 
choices, their figurative configuration can be deeply understood. Setting the detail level, the 
chromatic and luminous qualities, the quantity and the value of graphic signs in the VE, be- 
come essential features to define the sense of immersion and presence and, therefore, the 
communicative power driven by perceptual inputs. 
As Olga Bogdashina asserts, ‘filtering of an infinite amount of information is necessary to 
make the processing of information effective and conscious” [Olga 2015, p. | 10]. For this 
reason, the graphic design for an “autism-—friendly”VR has to provide essential information 
about the environment to reduce sensorial weight avoiding distraction. In fact, the repre- 
sentation path allows the process of “synthesis, communication and explicitation” of the 
space, necessary for decoding and subsequent learning its cultural contents. The relationship 
with environment can influence human abilities to the point of providing knowledge for 
non-neurotypical people as well as for anybody else. A right interpretation of spatial codes 
through drawing disciplines can provide a human-centered design of a virtual environment 
where architectural space becomes the medium for communication and inclusion. In future 
studies, the guidelines introduced in the present paper will be used to produce a prototype 
to test them with the user reference. 


Notes 


[I] So called referring to human optical system, elaborating images by projection on the retinal surface. 
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Abstract 


Learning through the direct experimentation of models, in their variety of manifestations and hybrid- 
izations that we know today, is undoubtedly a very powerful aid in the acquisition of knowledge. Spe- 
cifically on architectural form, this aid is even more important, because it helps not only to understand 
the form of existing architecture, but even more to imagine and design new ones. The research here 
presented, focusing on this double objective, cognitive and creative, proposes and experiments new 
ways of integrating and interacting with heterogeneous models — both physical and virtual — conceived 
for a scenario of musealization of the architectural form.A place where the user interacts and experi- 
ences the properties and peculiarities of form, in perceptive continuity between real and virtual space. 


Keywords 
descriptive geometry, form, digital model, augmented reality, projected augmented reality. 
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Fig. |. Hyperbolic 
paraboloid and its 
sections in the Los 
Manantiales Restaurant 
by Felix Candela (Mexico 
City, 1956). 


Fig. 2 Interactive model 
of the AR application for 
the exploration of the 
hyperbolic paraboloid 
properties. 


Therefore, the proposed models constitute an expositive prototype, with which to experiment 
the possibility of redesigning new ways of communicating the form that permit its three-dimen- 
sional exploration, revealing its peculiar characteristics. However, at the center of the exhibition 
project we do not find the final result, namely a surface or a curve, but the generative process 
that led to that particular spatial configuration, in other words its construction. The construction, 
understood as an existential demonstration of the form, is the foundation of the synthetic ap- 
proach, i.e. graphic approach, with which descriptive geometry operates, univocally characterizing 
the modus operandi of architects. 


Augmented Reality Experimentations 


The extreme simplicity and technological advances implemented in everyday tools, such as 
smartphones, tablets, laptops, have made it possible to spread AR-technologies in every level of 
education, from primary school to university. The advantage of not needing for additional hard- 
ware such as visors or helmets typical in the VR field, makes AR-technologies particularly suitable 
for applications in numerous and heterogeneous sectors of scientific and humanistic knowledge 
[Voronina et al. 2019]. Many studies show that AR plays a fundamental role in pedagogical appli- 
cations today, although their potential is still partially explored [Burton et al. 201 |, pp. 303-329 4; 
Wu et al. 2013, pp. 41-49]. 

As part of the project aimed at the realization of a museum of the form, the experimentation of 
AR is one of the principal models through which to experience the properties of the form and 
the effects that these transfer into the architectural project. This type of representation offers 
the possibility of direct interaction with the digital model, allowing to operate with the abstract 
entities typical of the geometry of space: one and two-dimensional forms, that otherwise could 
not be realized in the real world, can be controlled in a representation on the edge between 
virtual and physical reality. The user operates with the form in order to understand its properties, 
that reveal their evidence in the finished form but, even before, in the generative process that 
led to it. According to this double need of fruition, the application focuses its attention on the 
representation of two states of the form: tts construction process and its final configuration. The 
idea of representing generative processes and final configurations concerns both the pure geo- 
metric form and the one applied to the project. Depending on its geometric properties, this is 
declined in different ways, giving rise to a heterogeneous and morphologically varied repertoire 
of architectures, all referable to the same formal matrix. 
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Fig. 3 Geometrical 
genesis of the Palmira 
Chapel (Felix Candela, 
1958) starting by the 
hyperbolic paraboloid 
explored with the AR 
application. 


Fig. 4 Hybrid model of 
hyperbolic paraboloid 
through Projected AR 
application. 


ae 


From a strictly operational point of view, through the application the user can retrace the gen- 
erative process of construction of the pure geometrical form, recognizing the properties in its 
final configuration and understanding how these properties are reflected in the architectural 
form conditioning the aesthetic and constructive aspects. lt will therefore be possible to enjoy 
the model through the application by choosing the direction of the path to follow: from the 
geometric world to the architectural one or inferring the geometric properties of the form 
starting from its applications. This double way of reading favors the double objective: increasing 
awareness and ability regarding the geometric control of lines and surfaces feeding the education 
in geometry; increasing the knowledge by operating in space with the form, through a journey 
from the known to the unknown that leads to the derivation of new properties of figures [2]. 
The experimentation, still in progress, is oriented around the development of prototypal 
AR models, like that of the hyperbolic paraboloid and its applications (fig. 2). Developed in 
Unity 3D environment, the model is activated in virtual space through an image or a three— 
dimensional model (used as a Target), which provides the surface in different configurations 
resulting from its geometrical genesis (a skew quadrilateral or a saddle). This virtual surface 
can be explored by the user; who can classify it in different ways by combining its properties 
from a speculative geometric approach or from its applications in architecture. For example, 
from a synthetic point of view, this can be explored as a ruled surface whose generatrices 
and directrices are activated by contact with the surface itself. Otherwise, it can be consid- 
ered as a second-order algebraic surface of which to derive the axes and symmetry planes 
and, sliding in contact with it, remarkable sections generically oriented in space. Moreover, it 
can also be classified from the point of view of differential geometry by ranging the oscu- 
lating circles of principal curvatures. However, It is also possible to interact with the surface 
in question by sectioning it with notable planes, obtaining portions of surfaces that combine 
with each other, giving rise to a various morphological design repertoire, as in the case of 
some projects by Felix Candela (fig. 3). 

In addition to this AR model type, the experimentation foresees that shape analysis is also en- 
joyed through hybrid models Projected AR type, in which 3D prints of surfaces are augmented 
in their information content through the video projection of their remarkable properties (fig. 
4). In this case the physical model, reproduced with rapid prototyping techniques, is used for 
activating the AR projections, and for supporting projections themselves, which allow the user 
to learn the properties of the shape by directly interacting with the physical model. 
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Conclusions 


Representation has always had a central role in the genesis of the architectural project. 
The graphical exercise has always been an immediate expression of design thinking and 
a tool for the progressive elaboration of an idea that is nourished and grows through the 
repetition of its own representation, in a virtuous circle that accompanies the entire design 
process. Exploration of the form through drawing, namely by its construction, leads to increas- 
ing knowledge by passing from the known to the unknown, researching new properties of 
shape to derive. In this regard, the heuristic value of representation reveals its maximum ex- 
pressive potential. Thus, the three-dimensional representation and its exploration constitutes 
a flywheel for research into architectural form, revealing its geometrical motivations in virtual 
space. The proposed experimental models, developed through AR and Projected AR experi- 
ences, describe the construction of form and show its properties, illustrating the potential of 
aggregation deriving from them. Potentialities that generate a wide and heterogeneous mor 
phological repertoire of which several architectural projects are the expression. 


Notes 


[|] The project is inspired by the collections of mathematical models made in Europe between the second half of the nineteenth 
century and the early twentieth century. These exhibitions were aimed at “showing remarkable properties concerning the 
research topic investigated and showing some results that were progressively achieved in different fields of ‘pure’ and ‘applied’ 
mathematics: Descriptive and Projective Geometry, Analytical Geometry, Algebraic Geometry” [Palladino 2008, p. 31]. 


[2] The didactic purposes also include the implementation of learning tasks to verify understanding [Kaufmann 2003, pp. 339-345]. 


[3] The idea of knowledge as a passage from the known to the unknown is a founding principle of descriptive geometry. It was 
introduced by Monge in the first pages of his Geometrie Descriptive, where he illustrated its objectives and principles; today, in 
he field of digital representation, it still appears highly relevant [Monge 1798, p. 2]. 
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REPRESENTATION CHALLENGES 
Augmented Reality and Artificial Intelligence in 
Cultural Heritage and Innovative Design Domain 


AR&AI in the Didactics of the 
Representation Disciplines 


Roberta Spallone 
Valerio Palma 


Abstract 


The class of Digital Representation Techniques is an optional class for architecture and engineering 
students at the Politecnico di Torino. For the second year, the program of the class, focused on the the- 
oretical and applicative panorama of technologies for digital drawing, integrates an introduction to the 
most recent developments in augmented reality (AR) and artificial intelligence (Al), two technological 
fields that are deeply influencing the interpretation of physical assets through digital models. The class 
requires students to produce different deliverables, including a research essay, graphic representation 
documents (plates or brochures), a video proving animation and editing skills, and a prototype AR ap- 
plication. The proposed contribution presents methods, objectives, and some of the results of the class, 
highlighting the interaction between the different digital representation techniques and deepening the 
critical aspects related to the introduction of new AR and Al technologies. 


Keywords 
augmented reality, artificial intelligence, didactics, representation, design. 
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Introduction 


Al and AR, considered in their significant synergies, have only recently attracted the interest of 
scholars in the fields of design and enhancement of physical assets. The connection with digital 
acquisition and modelling technologies makes them ideal media for recognition, communication, 
and interaction with physical space. In the transdisciplinarity of complex processes that character- 
ize the design activity, the discipline of representation assumes a nodal role. On the one hand, in 
an increasingly effective way, digital models summarize the prerogatives of drawing and project 
enclosed in the Latin verb designo and in Alberti's lineamenta. On the other hand, the same mod- 
els, suitably simplified, become data archives capable of communicating the complexity of reality, 
increasing the share of transmissible information. 

Hence the importance of making these topics part of the research as well as of the teaching in 
the area of representation, as in the experience we have been conducting since last year in the 
Digital Representation Techniques class at the School of Architecture at the Politecnico di Torino. 
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Content and Experimentation of an Evolving Class 


The Digital Representation Techniques class was created in 2006 as an optional class for the 
Bachelor course in Architectural Sciences. The aim of the class is to provide students with 
an up-to-date theoretical and applicative overview of the new and evolving trends in digital 
representation for project development and communication. In view of the interest shown 
by students, with applications for more than the |50 places available, in 2013 the course was 
elected as an optional Athenaeum class, extending the offer to the courses of Design, Ter- 
ritorial Planning and Engineering (this last including around twenty different specializations). 
The student composition has become increasingly diverse over the years, with engineering 
students exceeding 35% in 2021. This has led to new stimuli for us to explore new subject 
areas, tailoring experiences to the students’ curricula. 
In the past and current academic year, the class, taught by Roberta Spallone and co—taught by 
Francesco Carota, Riccardo Covino and Valerio Palma, has aimed at incorporating the inputs 
offered by the most recent developments in Al and AR technologies. For this reason, Al for 
computer vision was introduced at a theoretical level through specific lectures held by the 
teachers and by invited scholars who deepened the relations between computer graphics, 
computer vision, human-machine interfaces, Al and also telecommunications and 5G in sup- 
port of virtual tourism. AR module has been developed at a theoretical and operational level 
under the supervision of Valerio Palma. 

The theoretical contents of the class focus on the representation and digital communication 
of the project, and are organized between photography in its relations with rendering, re- 
constructive modelling, representation in contests, product communication, animation, de- 
sign communication in websites. The operative contents include learning of rendering and 
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Fig. 2. AR exercises 

based on image tracking, 
developed in Unity and 
Vuforia. On the left: the 
object is superimposed 
against its design drawing. 
On the right: the user 
can start an animation 

to see the engine model 
disassembled; the figure 
shows the app preview 

in the Unity development 
interface. Authors: M. 
Ferraro, M. Marangone, 
D. Parente (left) E. Burati, 
A.L. Scardino, M.A. 
Valencia Zeballos (right). 


animation techniques, developed with Autodesk® 3ds Max and Blender software, video 
editing with DaVinci Resolve, and AR with the combination of the Unity® graphic engine and 
the Vuforia® Software Development Kit (SDK). 

The attention of the teaching team to free and open source software (FOSS) as well as to 
the possibility of carrying out the proposed experiments with computers and devices com- 
monly in use and at a low cost orient the pedagogical choices. 

In addition to an in-depth study of the theory, the practice requires groups of students of 
different backgrounds to work on the digital model of a project, which scope is defined by 
their specific skills (i.e. architectural and urban projects, product designs, mechanical compo- 
nents, characters animation, etc). 

The operations to be carried out on these models consist of rendering and animation, which 
flow into static graphic products such as plates or brochures, production of a video and, as 
will be discussed in more detail below, augmented reality experiences applied to the graphic 
works (fig. 1). The results of this last activity will be presented below, highlighting the interac- 
tion between different types of representation and digital communication. 


Insight Into the Module of Al&AR 


The module on Al and AR technologies aims at providing students with skills to deal with 
new digital tools and the ongoing changes in the use of information in representation and 
design. The way we structure our knowledge and the way in which the models become 
operational are deeply influenced by the possibility of producing and collecting an increasing 
amount of data [Batty 2016; Datta 2017]. The tools we bring to the students’ attention sup- 
port the understanding of physical space as a means for accessing information and ordering 
it. Both Al and AR technologies benefit from the growing computational power available and 
dedicated hardware and software on mobile devices. Al, which has undergone unprecedent- 
ed advancements in the last years, has found practical applications related to computer vision 
in many different fields [Goodfellow et al. 2016; Pezzica et al. 2019]. AR applications have 
spread over the past two decades, and related development tools constantly increase the 
range of trackable assets and the effectiveness of digital superimposition against real-time 
images [Amin, Govilkar 2015; Bekele et al. 2018]. The physical form of reality is a sort of mul- 
tidisciplinary platform, a shared grid (tangible and visible) on which to lay many information 
strata and different interpretations. Thus, the study of Al and AR allows us to reflect on the 
links between virtual and real, clarify the potential and limits of data, steer the tools towards 
more flexible, updatable, and scalable solutions. 

With this in mind, starting from experiences that mainly concern the architectural and urban 
scale, we try to stimulate an even wider interest, which may involve the different disciplinary 
paths of students attending the class. The program section dedicated to AR deals with the 
current development status of these technologies (both hardware and software), also by 
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Fig. 3. AR exercise based 
on image tracking, 
developed in Unity and 
Vuforia. The target image 
is printed on a mobile 
card to simulate an arm 
prosthesis. When the user 
reaches the virtual ball, 
the prosthetic hand grabs 
it. Authors: G. Brero, A. 
Porcelli, M. Pettiti. 


Fig, 4. AR exercise based 
on image tracking, 
developed in Unity and 
Vuforia. The students 
presented a videogame 
in which the AR 
target image has to be 
reassembled before to 

se the 3D layer. Then, 
the user is challenged 

to search the correct 
viewpoint to find an 
hidden message. Authors: 
E Rampolla, V. Mencarelli, 
D. Fiabane, M. Bosco. 


presenting the research experiences conducted at the Politecnico by the teachers. Besides, 
we try to clarify both the potential of the technology and the critical aspects (e.g., limits in in- 
teroperability). The theoretical part of the module deals with the most popular AR, Al tools 
for image recognition (in particular AR image tracking functions and DL algorithms) and 
integrates insights into the role of fifth-generation cellular networks (5G) in future scenarios 
of transmission of information. The aim is to emphasize that technological advances do not 
only constitute quantitative leaps in data gathering but also require the transformation of the 
methods and models to organize digital information. 


Objectives and Examples of the Students’ Work 


In the operational phase of the course (consisting of exercises) we provide the tools for 
building a simple AR application based on image recognition. We adopt the Unity video game 
development engine and the Vuforia AR kit, which are software with a short learning curve, 
requiring little previous experience with coding, and allowing free use for development. 
However, a broader landscape of competing products and development options is present- 
ed. Other practical indications are aimed at stimulating in-depth studies and experiments 
with particular AR or user interface functions, such as interactive elements, animations, oth- 
er possibilities related to the integration of scripts and non-flat AR markers. Beyond the 
technical requirements of the deliverables, students are expected to conduct the exercises 
reflecting on two fundamental and connected aspects. 

|. The first aspect is the link between real space and digital information (such as 3D mod- 
els, graphic interface elements, databases). We expect students not to consider the real 
object just as the target for AR activation but as an object that can be studied through 
its digital model with the help of AR. 

2. The second aspect that we ask to explore is the innovativeness of the developed applica- 
tion. In fact, in the case of digital representation, attention must be paid not to replicate 
uses of the model already available through other widespread technologies, such as vir- 
tual reality or simple on-screen navigation. 
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Therefore, we aim at continuity with the theoretical part, and we intend to highlight through 
practice the distinction between the excitement for novelty and more significant and long— 
lasting transformations enabled by technology. 

Among the first years results of this operational phase, several deliverables show good re- 
ception of the offered stimuli (the level of complexity depends on the students’ choice to 
deepen AR or other topics in the program). 
Some exercises emphasize the relationship between target image and model, letting the 
two parts complement each other, e.g., by using a printed drawing to show the constructive 
geometries of a design object while the AR model, correctly scaled, shows the rendering (fig. 
2, left). In other exercises with the simple image target system, students proposed interactive 
solutions, employing buttons and animations (fig. 2, right). 
There are also more elaborate works. A team of computer and biomedical engineering students 
worked on a robotic prosthetic arm project (started in another class). They produced an application 
prototype that features a wearable target to simulate the use of the prosthesis. The exercise also 
included the interaction with other virtual objects, e.g, for activating an animation of the arm (fig. 3). 
Another team of cinema and media engineering students has instead created a gaming appli- 
cation that features the interaction between a website and the mobile device, proposing an 
enigma in which AR is used to change the point of view on a three—dimensional object (fig. 4). 


Conclusion 


It is difficult to speak of conclusions for a class that is constantly evolving. Rather, by recording 
the success of the new topics in terms of interest and results, new transdisciplinary perspectives 
can be outlined, opening up the possibility of developing master's theses and doctoral courses, 
as is starting to happen. In fact, a thesis in Design and Visual Communication has just been 
discussed. It combined the disciplines of Representation and Information Processing Systems in 
the realization of AR and VR experiences for the enhancement of the heritage of the Museum 
of Oriental Art (MAO) in Turin, and two more on the same topics are being prepared [1]. 


Notes 


[1] This paper, the result of a teaching activity shared by the two authors, was written by: R. Spallone (Introduction, Content and experi- 
mentation of an evolving class, Conclusion) V. Palma (Insight into the module of AI&AR, Objectives and examples of the students’ work). 
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Abstract 


This paper aims to test algorithms for 3D reconstruction from a single image specifically for building 
envelopes. This research shows the current limitations of these approaches when applied to classes 
outside of the initial distribution. We tested solutions with differentiable rendering, implicit functions, 
and other end-to-end geometric deep learning approaches. We recognize the importance of gener- 
ating a 3D reconstruction from a single image for many different industries, not only for Architecture, 
Engineering, and Construction (AEC) industry but also for robotics, autonomous driving, gaming, 
virtual and augmented reality, drone delivery, 3D authoring, improving 2D recognition and many 
others. Henceforth, engineers and computer scientists could benefit, not only from having the 3D 
representations but also from the Building Information Model (BIM) at their disposal. With further 
development of these algorithms it could be possible to access specific properties such as thermal, 
physical, maintenance, cost, and other parameters embedded in the class. 


Keywords 
geometric deep learning, monocular 3D reconstruction, building envelope, architecture. 
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Introduction 


Currently, we mainly capture reality around us with static 2D media, such as pictures, or 
videos, in case we add the temporal component to it. For instance, architects represent their 
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Fig, |. Original from 
Facebook MeshRCNN — 
Adaptation to 
Architectural Field. 
(Testing Mesh-RCNN on 
the pictures of building 
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Related Work 


Methods for 3D reconstruction from single-image are complicated by the fact there could 
be many possible reconstructions when the object is not entirely visible; hence, most of them 
need to rely on strong supervision. Therefore, they use datasets such as ShapeNet or Mod- 
elNet [Wu et al. 2015] Other methods learn from images paired with aligned 3D meshes or 
require keypoint annotations on the 2D training images [Wu et al. 2016] and/or multiple views 
for each object instance, often with pose annotations. Shading becomes an important cue for 
3D understanding, explored in numerous works over the years [Henderson et al. 2020]. Dif- 
ferent methods have been explored in the past: mesh based such as N3MR [Kato et al. 2018], 
or voxel based like 3D-R2N2 [Choy et al. 2016] and MVD [Smith et al. 2018], or point based 
like PSG [Fan et al. 2017] and many others [Aubry et al. 2014]. These have issues in performing 
a complete task with objects not within the training distribution, so we wanted to confirm our 
hypothesis and stress these limitations [Henderson et al. 2020;Wang et al. 2019]. 


3D Reconstruction From a Single Image 


Learning—based 3D reconstruction works are based on different 3D representations as pre- 
sented before. While voxel representations prove to be computationally expensive, point 
cloud representations are demonstrated to be rotation and translation invariant, and compu- 
tationally more efficient than voxels [Liu et al. 2019]. Moreover, mesh representations, better 
preserve the connections between distinct parts and are more suitable for fine-grain detailed 
representations. Modern implicit functions not only prove to be extremely efficient with their 
continuous and differentiable representation of the iso-surface with a binary value indicating 
whether a point is within the volume, but also more accurate for tasks such as reconstruction 
and 3D shape completion [Gu et al. 2020]. Nerf, Occupancy Network, DeftTef [Gao et al. 
2020] have recently followed for this task. 

Within the AEC, 3D shapes and objects preserve a common grammar and they are com- 
posed by a fixed set of components such as windows, doors, roof, floors, walls, and others. 
While the typology can change, the main elements in the building stay the same for most 
cases (except for some iconic buildings and pavilions). The philosophy of Hoffman and Rich- 
ards influenced this research. In fact, they viewed object recognition tasks as a visual system 
decomposition of shapes into parts with their descriptions and spatial relations. In the same 
way, we propose that the best way of representing a building reconstruction is to assemble 
each component together, orienting their quaternions to perfectly fit an initial picture which 
was inspired by the CompoNet work [Schor et al. 2019]. In contrast to the approach, we aim 
to translate the assembly algorithm, specifically for an an architectural task. They used a gener- 
ative neural network for generating 3D shapes from a 2D image, based on a part-based prior, 
where the key idea was for the network to synthesize shapes by varying both the shape parts 
and their compositions. Treating a shape not as an unstructured whole, but as a composable 
set of deformable parts, adds a combinatorial dimension to the generative process to enrich 
the diversity of the output, encouraging the generator to venture more into the “unseen”. 
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They generated a plethora of shapes compared with baseline generative models using their 
custom metrics. The assembly—based synthesis was inspired by 3D shape assembly research 
that generates new shapes from a combination of various parts [Huang et al. 2020; Li et al. 
2019] from a single image. 


Conclusion 


We saw that projects such as Mesh-RCNN lack the ability to perform well with unseen 
classes. This limitation of generalizing to unseen classes make these approaches challeng- 
ing to adopt. Furthermore, the training of these algorithms required multi-GPU training(8 
GPUs V 100, for Mesh-RCNN) that not all the researchers can access. The current lack of a 
common balanced dataset (with intra and inter-class variance), or pre-trained models that 
generalize well to unseen data, are missing in the research community, and with this research 
we hope to stress the importance of the creation of such datasets and models. Another 
limitation is embedded in the metrics used to evaluate the performance of these algorithms: 
chamfer distance, EMD (earth moving distance), mAP and others offer good quantitative 
results distant from a recognizable representation that follows qualitative results. Finally, the 
creation of such dataset could provide new research on 3D shape explorations for archi- 
tects using Generative Adversarial Network in 3D like ShapeGAN and 3DGAN [Kleine- 
berg et al. 2020; Freeman et al. 2016]. 
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